Not sure if I have this right but this is how I understand it
> So is the magic here that it's Postgres? What makes being able to query something in Postgres special?
There are a bunch of pros and cons to using Postgres vs. DuckDB. The basic difference is OLTP vs. OLAP. It seems pg_lake aims to give you the best of both. You can combine analytics queries with transactional queries.
pg_lake also stores and manages the Iceberg catalog. If you use DuckDB you'll need to have an external catalog to get the same guarantees.
I think if you're someone who was happy using Postgres, but had to explore alternatives like DuckDB because Postgres couldn't meet your OLAP needs, a solution like pg_lake would make your life a lot simpler. Instead of deploying a whole new OLAP system, you basically just install this extension and create the tables you want OLAP performance from with `create table ... using iceberg`
> when we say it’s now queryable by Postgres, does this mean that it takes that data and stores it in your PG db?
Postgres basically stores pointers to the data in S3. These pointers are in the Iceberg catalog that pg_lake manages. The tables managed by pg_lake are special tables defined with `create table ... using iceberg` which stores the data in Iceberg/Parquet files on S3 and executes queries partially with the DuckDB engine and partially with the Postgres engine.
It looks like there is good support for copying between the Iceberg/DuckDB/Parquet world and the traditional Postgres world.
> Or it remains in S3 and this is a translation layer for querying with PG?
Yes I think that's right -- things stay in S3 and there is a translation layer so Postgres can use DuckDB to interact with the Iceberg tables on S3. If you're updating a table created with `create table ... using iceberg`, I think all the data remains in S3 and is stored in Parquet files, safely/transactionally managed via the Iceberg format.
and don't forget that Grok is powered by illegal cancer-causing methane gas turbines in a predominantly black neighborhood of Memphis that already had poor air quality to begin with
There are a lot of myths about the way humans used to be, especially Native Americans. Were they utopian nature-lovers? Were they barbaric human-sacrificers?
A good book on this topic is The Dawn of Everything, written by an anthropologist and an archaeologist. A YouTube video from one of the authors is here https://www.youtube.com/watch?v=8SJi0sHrEI4
I disagree with the idea that "barbarism was common" in Native American societies. I don't think you can generalize from the Incas so directly like this
Lemmy has been good for me, though the population is still small.
This can be either a good thing or a bad thing depending on what you are looking for.
I've had a similar experience with Bluesky, though it's much more like Twitter than Hacker News. You can curate a good feed by following a ton of people, then unfollow the noisy ones as you look over the feed. You can use "starter packs" and hashtags to help get started, too.
Once you've found some people you like, this tool is somewhat helpful for finding more people you might like to follow:
I totally get why so many peple are jumping to BlueSky, and I may yet, one day, join them. But I'm still almost exclusively on Mathstodon.xyz, only visiting a few places to stay in touch with a few people who are refusing to move (sometimes for perfectly valid reasons).
Having said that, I invite you to read this essay:
For people who build connections and create content, I advise most strongly that they prepare from day one for having to move. Create your content elsewhere and then copy it to BS, keep a separate record of people you want to stay in touch with. Do not rely on a platform like BS for keeping anything.
The article exactly captures why I am using Mastodon-the-Platform and not BS. If I do put things on BS, I will be prepared to lose them, and connections.
But each person has their own reasons for using any of these platforms. I'm getting great technical content, mathematical content, connections, and conversations on Mathstodon.
PS: For the avoidance of doubt: mathstodon.xyz (carefully note the spelling) is one instance of Mastodon-the-Platform.
PPS: Again, I totally get why people are choosing other platforms, and this is not a criticism. It purely an explanation of my reasoning.
I noticed from the docker overlay filesystem that the container was spraying files all over the disk. (Ephemeral, destroyed on container shutdown, sure, but I wanted to reduce write-wear on my ssd...)
I tried setting it up with /tmp as a tmpfs (ramdisk) but it then refused to start...
Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?
Modern SSDs are pretty good at things like wear levelling.
For example [1] reports that a bunch of 256 GB SSDs lasted to 2000+ terabytes written, and a handful up to 7000 terabytes written. So you could saturate a 100 megabit internet connection for 5 years before even a small SSD would wear out. And an SSD 4x the size has 4x the life.
If you're running on a raspberry pi with a microsd card for storage, feel free to keep worrying though :)
> the container was spraying files all over the disk
Right, that's basically the point...the Warrior downloads files, compresses them, and uploads them for archival. This necessarily requires staging the files somewhere between download and upload.
> Anyone know any broad-spectrum docker incantations to force all overlay writes to RAM, for a container?
Why would you want this? This sounds like a terrible footgun.
You can put the entire docker directory in a ramdisk. Same as you would when trying to move it to a secondary harddisk. Risky though as a reboot would wipe everything
I don't think the "noble savage" idea applies in this case. The problem of the noble savage idea is that it portrays indigenous people as simple, pure, and uncorrupted, while overlooking that indigenous people are complex just like non-indigenous people.
It's no myth that the Haudenosaunee and other indigenous people had sophisticated governments that could have inspired the writers of the US Constitution. The Haudenosaunee's democracy-ish form of government extends back probably a thousand years. The people who wrote the US Constitution had contact with these people. The exact extent to which this shaped the Constitution is up for debate, of course.
Yes, the US government draws from European roots too. I hope my kids learn about both the Magna Carta and the Great Law of Peace.
> So is the magic here that it's Postgres? What makes being able to query something in Postgres special?
There are a bunch of pros and cons to using Postgres vs. DuckDB. The basic difference is OLTP vs. OLAP. It seems pg_lake aims to give you the best of both. You can combine analytics queries with transactional queries.
pg_lake also stores and manages the Iceberg catalog. If you use DuckDB you'll need to have an external catalog to get the same guarantees.
I think if you're someone who was happy using Postgres, but had to explore alternatives like DuckDB because Postgres couldn't meet your OLAP needs, a solution like pg_lake would make your life a lot simpler. Instead of deploying a whole new OLAP system, you basically just install this extension and create the tables you want OLAP performance from with `create table ... using iceberg`
> when we say it’s now queryable by Postgres, does this mean that it takes that data and stores it in your PG db?
Postgres basically stores pointers to the data in S3. These pointers are in the Iceberg catalog that pg_lake manages. The tables managed by pg_lake are special tables defined with `create table ... using iceberg` which stores the data in Iceberg/Parquet files on S3 and executes queries partially with the DuckDB engine and partially with the Postgres engine.
It looks like there is good support for copying between the Iceberg/DuckDB/Parquet world and the traditional Postgres world.
> Or it remains in S3 and this is a translation layer for querying with PG?
Yes I think that's right -- things stay in S3 and there is a translation layer so Postgres can use DuckDB to interact with the Iceberg tables on S3. If you're updating a table created with `create table ... using iceberg`, I think all the data remains in S3 and is stored in Parquet files, safely/transactionally managed via the Iceberg format.
https://github.com/Snowflake-Labs/pg_lake/blob/main/docs/ice...