Polars also has all of these benefits (to some degree), but also allows for larger-than-memory datasets. Also has GPU backend, distributed backend, etc. Polars is heavily underrated, even with the recent hype.

I downvoted all your recs for polars, 1 because this is a DuckDB thread and it's low-key rude, and 2 because there are 4 of them. I wouldn't have minded if there were a single post that were like "DuckDB is cool, polars could be an alternative if..."

DuckDB has this capability as well: https://duckdb.org/docs/stable/guides/performance/how_to_tun...

Interesting, I wasn't aware; thanks for that. I will say, Polars' implementation is much more centered on out-of-core processing, and bypasses some of DuckDB's limitations ("DuckDB cannot yet offload some complex intermediate aggregate states to disk"). Both incredible pieces of software.

To expand on this, Polars' `LazyFrame` implementation allows for simple addition of new backends like GPU, streaming, and now distributed computing (though it's currently locked to a vendor). The DuckDB codebase just doesn't have this flexibility, though there are ways to get it to run on GPU using external software.

Thanks for that insight as well! My needs don't tend to be so demanding so I've gotten away without knowing these details, but I suspect I the not-so-distant future this could be useful to know.

Being able to use distributed backends to process frames sounds kind of incredible, but I can't imagine my little projects ever making use of it. Still, very cool stuff.

Have you seen Ibis[1]? It's a dataframe API that translates calls to it into various backends, including Polars and DuckDB. I've messed around with it a little for cases where data engineering transforms had to use pyspark but I wanted to do exploratory analysis in an environment that didn't have pyspark.

[1] https://ibis-project.org/

Polars has all of these benefits (to some degree), but also allows for larger-than-memory datasets.

Polars has all of these benefits (to some degree), but also allows for larger-than-memory datasets.

DuckDB supports this as well, depending on which benchmark you look at it regularly performs better on those datasets than Polars.

They will get away with it if we believe we are powerless to change it. Russia has been proven to be pushing defeatist propaganda similar to your sentiment, and I'm sure Israel has been as well.

Are Gazans receiving adequate nutrition? If not, why are we discussing semantics?

Starving infants is justifiable sometimes? When?

It is justifiable to stop a shipment of baby formula if that baby formula is known to be unsafe and carry bacteria that will kill infants.

I think in this particular case it's quite safe to say that those blocking the shipments aren't acting in good faith, however.

Does the food carry bacteria in reality? Why are we talking about bizarre hypotheticals?

Starving infants is morally justifiable if it's possible to make a rocket from their food?

Preventing the entry of something that can be made into a weapon is justified, yes. If you want to call that “starving children”, that’s up to you.

Withholding food from children results in children starving. It’s not semantics.

A Pew poll in 2022 found that about half (48%) of Israeli Jews agree that Arabs should be expelled or transferred from Israel. Horrifying statistic.

Now do Arab polls.

Seriously, you can't apply western standards to middle eastern coumtry, even more so, single out one ME country to apply them to.

Being against genocide isn’t a western standard.

1) Not a genocide

2) If forces were reversed, it would be one for sure.

--

Anyhow, war and terror is on the rise in Europe right now, cue immediate swing to far right in the polls, so good luck!

I thought the opposite - they set a precedent indicating that reproduction of a copyrighted text by an LLM is infringement. If authors refuse to sell to them (via legal terms indicating LLMs aren't allowed), it's infringement. No?

I'd be curious to hear from a legal professional...