Fast Dataframes for Big Problems
I was recently in a secret demo run by the Cuda and Polars team. They passed me through a metal detector, put a bag over my head, and drove me to a shack in the woods of rural France. They took my phone, wallet, and passport to ensure I wouldn’t spill the beans before finally showing off what they’ve been working on.
Or, that’s what it felt like. In reality it was a zoom meeting where they politely asked me not to say anything until a specified time, but as a tech writer the mystery had me feeling a little like James Bond.
In this article we’ll discuss the content of that meeting: a new execution engine in Polars that enables GPU accelerated computation, allowing for interactive data manipulation of 100GB+ of data. We’ll discuss what a data frame is in polars, how GPU acceleration works with polars dataframes, and how much of a boost to performance one can expect with the new CUDA powered execution engine.
Who is this useful for? Anyone who works with data and wants to work faster.
How advanced is this post? This post contains simple but cutting-edge data engineering concepts. It’s relevant to readers of all levels.
