Using Large Datasets with pandas

Using Large Datasets with pandas

37mIntermediate2024-02-13

Authors

Miki Tebeka

Miki Tebeka

CEO at 353Solutions

Course details

As data grows in size and complexity, most enterprises start to think about how to migrate to a larger-format data system such as Spark. However, this move can be quite painful, and you’ll most likely need to learn an entirely new set of tools. In this course, join instructor Miki Tebeka to learn how to get started working with large datasets using pandas, the fast, powerful, flexible, and easy-to-use data analysis tool built on top of the Python programming language. Find out how to navigate storage formats, tips for saving memory, efficient memory computation strategies, and more. Along the way, Miki also demonstrates how to leverage a handful of alternatives to pandas that still use the same API, such as Dask, Polars, and Beefy VM.

Skills covered

pandasData EngineeringData AnalysisData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceOne-Off

Concepts

Introduction

  • Larget datasets with pandas
  • What you should know
  • Setting up

Big Data

  • The problem with big data
  • Big data is dead
  • Big data systems

Using Less Memory

  • Load only what you need
  • Types
  • Categorical data
  • Arrow
  • Challenge - Daily passengers
  • Solution - Daily passengers

Memory Efficient Computation

  • Iterative computation
  • SQL
  • Faster calculations
  • Challenge - Maximum passengers
  • Solution - Maximum passengers

Other Options

  • dask
  • polars
  • Beefy VM
  • Challenge - Median ride distance
  • Solution - Median ride distance

Conclusion

  • Next steps
40,000 Toman