// HACKER NEWS — CYBERSECURITY
Pandas Should Go Extinct
This post covers material from a talk I gave at Latency Conference. If you want to just:
You read that correctly, Pandas should go extinct. Not the cute fluffy things used for international diplomacy, but the Python DataFrame library.
Why? Because Pandas’ inefficiencies force you to adopt distributed querying systems before your workloads justify the added complexity. I posit that most workloads will never justify those systems, they are just well marketed “silver bullets”.
To understand what I’m talking about we first must understand the typical adoption pathway for Pandas.
The diagram below shows a rough guide of when you typically would consider adopting a given DataFrame library based on the data size you are working with. Following it from left to right, you also see the typical adoption pathway for data analysis tools, and the cliff that Pandas’ users experience beyond a certain data size.
People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API. The traditional answer at this point is to graduate to a “real” (read: expensive) tool like Spark, DataBricks, Snowflake, or Dask designed for Big Data ™️
Here’s the thing: there’s a growing gap between the “Pandas cliff” and the scale where distributed systems are genuinely necessary. This gap, sits somewhere around the 100GB mark, and can be effectively filled by modern, high-performance, single-machine tools. I’m primarily talking about Polars and DuckDB.
Why do we care so much about this ~100GB threshold? The answer lies in understanding how much “Big Data” exists in the wild.
In 2024, Amazon published a paper entitled “Why TPC is not enough: An analysis of the Amazon Redshift fleet”. The aim of this paper was to compare telemetry data from Amazon’s own distributed analytics database, Redshift, with the query patterns used in industry standard database benchmarks. As part of their analysis Amazon published fleet statistics on query run times and table sizes.
If we’re willing to make a couple of assumptions we draw some interesting conclusions about how Amazon’s customers are using analytics databases. Let’s assume that: