Your billing partner is AWS. Polars' markup is on your AWS bill.

Hi, I am the original author and CEO of Polars. We are not focused on SQL at this time and provide a DataFrame native API.

Polars cloud will for the moment only support our DataFrame API. SQL might come later on the roadmap, but since this market is very saturated, we don't feel there is much need there.

Polars | Rust Engineer | Amsterdam ONSITE / HYBRID

At Polars we're building a fast distributed query engine for Polars DataFrames. Our mission is scale DataFrame processing and offer a modern verstatile API to process data fast and easy.

https://hiring.pola.rs/o/database-engineer

Polars | Rust engineers | DevOps | FrontEnd | Hybrid

Polars is built on the foundation of a vibrant and active open-source community, and we embrace that philosophy in how we run our company. We trust talented people to do their best work without unnecessary constraints. Collaboration is key, but we keep meetings to a minimum to maintain focus. As Polars and Polars Cloud continue to set a new standard in Python data processing, we're looking for like-minded individuals to join us on this journey.

See more: https://pola.rs/careers/#positions

import polars.col as C

C.blah

Thanks! I'm not sure if pl.col improved since the last time I looked at polars or if I was too lazy to find it, but pl.col (docs) look great!

Disclosure, I am the author of Polars and this post. The difference with Ibis is that Polars cloud will also manage hardware. It is similar to Modal in that sense. You don't have to have a running cluster to fire a remote query.

The other is that we are only focussing on Polars and honor the Polars semantics and data model. Switching backends via Ibis doesn't honor this, as many architectures have different semantics regarding NaNs, missing data, order of them, decimal arithmetic behavior, regex engines, type upcasting, overflowing, etc.

And lastly, we will ensure it works seamlessly with the Polars landscape, that means that Polars Plugins and IO plugins will also be first class citizens.

It’s funny you mention Modal. I use modal to do fan-out processing of large-ish datasets. Right now I store the transient data in duckdb on modal, using polars (and sometimes ibis) as my api of choice.

I did this, rather than use snowflake, because our custom python “user defined functions” that process the data are not deployable on snowflake out of the gate, and the ergonomics of shipping custom code to modal are great, so I’m willing to pay a bit more complexity to ship data to modal in exchange for these great dev ergonomics.

All of that is to say: what does it look like to have custom python code running on my polars cloud in a distributed fashion? Is that a solved problem?

Yes, you can run

`pc.remote(my_udf, schema)`

Where

`def my_udf() -> DataFrame`

We link the appropiate Python version at cluster startup.

The cluster runs in your own VPC.

Disclosure, I wrote this post. The compute-plane (cluster) will run in your own VPC.

Nothing. Polars -> pola.rs

The Polars name and a hint to the .rs file extension.

I’m aware. I personally wouldn’t want to tie my infrastructure, nor provide funding, to the government of Serbia at this particular juncture in geopolitical time, but hey, you gotta have a cutesy ccTLD hack or you aren’t webscale.

I do think code should be shared when you are benchmarking. He could be using Polars' eager API for instance, which would not be apples to apples.

Hi - Miles Cole here… I used lazy APIs where available. I.e. everything up to write_delta() is lazy in the Polars (Mod) variant.

Yeah I was debating whether to share all of the source code. I may share a portion of it soon.

Great! A small correction on your post. Polars does have SQL suppor. It isn't the main usecase, so it isn't as good as that of Spark and DuckDB, but it does exist and is being improved on.

Ritchie - thx for graciously correcting some things I got wrong, will get it corrected!