I'm not following the trends closely, but has Polars become a full replacement for Pandas? Are there use cases where one is better suited than the other?
> 1 Billion Row Challenge benchmark: Pandas took 4m28s vs. Polars 5.04s and DuckDB 5.19s — DuckDB also used 19x less memory
Python Vs Rust : In terms for speed - No comparison
(The above episode transcript has a link to blog post titled "Pandas should go extinct" )
esco2292 3 hours ago [-]
Polars is effectively a full replacement for Pandas for 99.9% of all cases. The only exception I'm really aware of is if you're working with geospatial data, as there isn't yet a "Geopolars" equivalent of the commonly used "Geopandas". However, Geopolars is still in active development and should eventually be production ready.
adeptima 2 hours ago [-]
it’s on the correct path. i use rust for geo spatial and the gap with c, c++ closing rapidly or negligible in most cases
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
Yes and no, its not replacing the reason why pandas was popular ie data scientists, but it a full replacement of its pipeline usage, And I would saw also beating out spark
seemaze 29 minutes ago [-]
It has been for me. I greatly prefer the API, it fits my mental model much better. Give it a try!
392 3 hours ago [-]
my understanding is Polars is faster, scales better without using external solutions, better API, +Rust. Pandas wins if you want to use what the vast majority of folks are using and have used in the past. Probably has a more complete set of helpers / recipes for the little things you bump into when using it thoroughly, but in the age of LLMs, I think that's minor.
vovavili 2 hours ago [-]
>Pandas wins if you want to use what the vast majority of folks are using
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
derriz 1 hours ago [-]
It’s the API that gave me the push to leave Pandas. 10 or more years of occasional Pandas use and I still had to google for any non-trivial queries.
In that regard, I’m still waiting for a credible jq replacement…
boltzmann64 11 minutes ago [-]
Learn SQL and interface with Duck. You will be 100x faster than Pandas/Polaris duo at fraction of memory. Also SQL is supported literally everywhere with a much more capable than Pandas API. Duck outputs to a Pandas Dataframe, but just treat that like a dictionary. Do all your processing, filtering and aggregation in Duck.
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
mhh__ 45 minutes ago [-]
Avoiding pandas developers is a great reason to use polars imo
There are awkward things. For example, if you ingest a nanosecond resolution timestamp, there's no way to re-export that out of the Polars dataframe with nanosecond resolution.
therno 2 hours ago [-]
I use Polars 2.0(rc) to (pre)calculate billions of weather scores on https://therno.com and it has been a lifesaver
Happy that I can upgrade to 2.0 final tonight.
Centigonal 12 minutes ago [-]
out of core sounds awesome! biggest thing that forced me to switch from pandas/polars to other solutions back in the day.
niltecedu 1 hours ago [-]
A bit surprised about the datafusion results from the post, I have tried it time and time again, but datafusion has always been the leading/trading blowers with polars for our workfloads with duckdb being vastly slower.
Doing the benchmarks for 2.0 on the large AWS metal machines at small data sizes (SF=10) really opened my eyes that we have some low-hanging fruit in Polars when it comes to optimizing our constant overhead for smaller queries.
For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.
I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.
sgarland 3 hours ago [-]
TIL that Polars supports SQL. Amazing.
brap 41 minutes ago [-]
At what point can we say that Polars is basically an in-memory database? (Genuine question)
dist-epoch 55 minutes ago [-]
It did since the first releases, but it was limited.
Now it seems they want to go head to head with DuckDB.
efromvt 42 minutes ago [-]
Love competition in the local data SQL space, makes everyone better
dkgs 3 hours ago [-]
A coincidence with the fact duckdb is supposed to release 2.0.0 very soon? :)
orlp 3 hours ago [-]
Actually yes. We had already been planning to do 2.0 for a long time. We originally said we'd move on from 1.x quickly when released 1.0 but ended up staying at 1.x much longer than intended.
From a quick check our first PRs were merged to the 2.0 branch in June:
2026-06-17T21:27:51Z #27993 chore: Stop coercing `pl.col(...)` to selector ...
2026-06-18T14:19:26Z #27996 chore!: Replace multi-seed hash API with a single seed
2026-06-19T07:05:11Z #27991 chore(python!): Remove `Expr.flatten` function
m00dy 3 hours ago [-]
Are they competing for something?
vindex10 3 hours ago [-]
> first class SQL support, which together with the performance improvements has Polars leading DataFusion and DuckDB in TPC-H and TPC-DS1 benchmarks,
Apparently about something ))
hnd9q09qk4 3 hours ago [-]
Thing I care about most is whether the old eager-vs-lazy footguns got cleaned up. Half my bugs were a stray collect() in a loop killing the query plan.
dartharva 2 hours ago [-]
Been using polars for over a year now, it is fantastic.
Kinrany 3 hours ago [-]
How does Polars relate to DataFusion these days? There's no reason for them not to converge into a single ecosystem, is there?
I'm also using it for BlockRotate, it's time to upgrade.
mrtimo 1 hours ago [-]
I can use pandas to clean a dataset, but each cleaning task is usually one line of code. OTOH, With DuckDB with one SQL statement I can replace 40+ lines of polars/pandas. You may reply, SQL isn't as easy to understand! Fair point, it's a declarative language... which is why I use Malloy. Malloy is to TypeScript as Javascript is to SQL. Malloy is much easier to read and write (just as TypeScript is) because it has a built in semantic model -- all the joins, measures, and dimensions are done in one place.
Here is an example [1] of visualizing college football games. Here are all the queries, and semantic model that power all the visualizations [2] Here is the AI generated typescript/react that does the visualizations [3]. The Malloy ecosystem has Malloyyo and Publisher which are replacements for PowerBI and Tableau and Looker. Here is another example for visualizing global trade [4].
> 1 Billion Row Challenge benchmark: Pandas took 4m28s vs. Polars 5.04s and DuckDB 5.19s — DuckDB also used 19x less memory
Python Vs Rust : In terms for speed - No comparison
(The above episode transcript has a link to blog post titled "Pandas should go extinct" )
from https://github.com/pola-rs/geopolars/tree/main
Comparison with GeoPandas
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
It's been impressive!
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
In that regard, I’m still waiting for a credible jq replacement…
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
tl;dr yes
Happy that I can upgrade to 2.0 final tonight.
For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.
I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.
Now it seems they want to go head to head with DuckDB.
From a quick check our first PRs were merged to the 2.0 branch in June:
Apparently about something ))
cant wait to upgrade my Quant trading bot
Here is an example [1] of visualizing college football games. Here are all the queries, and semantic model that power all the visualizations [2] Here is the AI generated typescript/react that does the visualizations [3]. The Malloy ecosystem has Malloyyo and Publisher which are replacements for PowerBI and Tableau and Looker. Here is another example for visualizing global trade [4].
[1] - https://mrtimo.github.io/cfb-games/games-2026.html?week=Week... [2] - https://github.com/mrtimo/cfb-games/blob/main/drives.malloy [3] - https://github.com/mrtimo/cfb-games/blob/main/dashboards/gam... [4] - https://tradeexplorer.org/