“The Party told you to reject the evidence of your eyes and ears. It was their final, most essential command.”
“Every record has been destroyed or falsified, every book rewritten, every picture has been repainted, every statue and street building has been renamed, every date has been altered. And the process is continuing day by day and minute by minute. History has stopped. Nothing exists except an endless present in which the Party is always right.”
― George Orwell, 1984 (2026?)
Just wait until the AI "layer" gets fast enough to rewrite the web in real time. Text, Photos, Videos, even real time phone calls will soon be in the grasp of the corporations. Forever locking us into our own personal prisons, controlled silos of information perfectly crafted and tailored to extract the maximum value where truth is not just hard to know but is imposable to know.
My original comment was just a counterpoint to the doom in parent: not all is lost, and in fact, quite a lot is not.
The situation of “things we care about are far away and require intermediaries to connect with” and “our ability to trust intermediaries is gone” are both human creations, and totally addressable.
> »This has even boosted efficiency to almost 27%, which represents the state-of-the-art. After 1,200 hours of continuous operation under standard illumination, no decrease in efficiency was observed.«
Kim, M. et al. Conformal quantum dot–SnO2 layers as electron transporters for efficient perovskite solar cells. Science 375, 302–306 (2022). https://doi.org/10.1038/s41566-025-01791-1
DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus.
DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handle >100k single row inserts/sec.
Transactions don't come for free. Embedding the engine in the catalog rather than the catalog in the engine enables transactions across analytical and operational tables. That way you can do a very high rate of writes in a heap table, and transactionally move data into an Iceberg table.
Postgres also has a more natural persistence & continuous processing story, so you can set up pg_cron jobs and use PL/pgSQL (with heap tables for bookkeeping) to do orchestration.
There's also the interoperability aspect of Iceberg being supported by other query engines.
How does this compare to https://www.mooncake.dev/pgmooncake? It seems there are several projects like this now, with each taking a slightly different approach optimized for different use cases?
I think pg_mooncake is still relatively early stage.
There's a degree of maturity to pg_lake resulting from our team's experience working on extensions like Citus, pg_documentdb, pg_cron, and many others in the past.
For instance, in pg_lake all SQL features and transactions just work, the hybrid query engine can delegate different fragments of the query into DuckDB if the whole query cannot be handled, and having a robust DuckDB integration with a single DuckDB instance (rather than 1 per session) in a separate server process helps make it production-ready. It is used in heavy production workloads already.
No compromise on Postgres features is especially hard to achieve, but after a decade of trying to get there with Citus, we knew we had to get that right from day 1.
Basically, we could speed run this thing into a comprehensive, production-ready solution. I think others will catch up, but we're not sitting still either. :)
> For instance, it can handle >100k single row inserts/sec.
DuckLake already has data-inlining for the DuckDB catalog, seems this will be possible once it's supported in the pg catalog.
> Postgres also has a more natural persistence & continuous processing story, so you can set up pg_cron jobs and use PL/pgSQL (with heap tables for bookkeeping) to do orchestration.
This is true, but it's not clear where I'd use this in practice. e.g. if I need to run a complex ETL job, I probably wouldn't do it in pg_cron.
select cron.schedule('flush-queue', '* * * * *', $$
with new_rows as (
delete from measurements_staging returning *
)
insert into measurements select * from new_rows;
$$);
The "continuous ETL" process the GP is talking about would be exactly this kind of thing, and just as trivial. (In fact it would be this exact same code, just with your mental model flipped around from "promoting data from a staging table into a canonical iceberg table" to "evicting data from a canonical table into a historical-archive table".)
Not to mention one of my favorite tools for adding a postgres db to your backend service: PostgREST. Insanely powerful DB introspection and automatic REST endpoint. Pretty good performance too!
What does data frames mean in this context? I'm used to them in spark or pandas but does this relate to something in how duckDB operates or is it something else?
i have so desperately wanted to love and use ducklake, but have come across some issues with it in practice (pg catalog). they seem to have to do with maintenance activities and ducklake suddenly throwing http/400 errors on files it created. i’m not sure if it’s due to my write patterns (gather data from sources into a polars dataframe and insert into the ducklake table from the df) into the partitioned tables or something else.
it’s ok in dev/test and for me as the person in the team who’s enamored with duckdb, but it’s made the team experience challenging and so i’ve just kinda reverted to hive partitioned parquet files with a duckdb file that has views created on top of the parquet. attach that file as read only and query away.
i may work up a full example to submit as an issue but up until now too may other things are dominating my time.
Base load becomes very expensive under free and fair market conditions. The reason is simple: wind and PV are extremely cheap, and surplus capacity costs little. PV module price in EU is just 0.086 UDS/W fob. Wind turbine price in China ~2200Yuan/kW inclusive tower.
In a free market, this leads to attractive conditions for batteries, and that is where the problem of base load begins: there is a lack of real demand, and base load then remains unused because its OPEX cannot compete with wind and PV.
There's just one problem. There is virtually no free and fair market in the electricity market. Utilities lobby very successfully for highly regulated markets to protect their monopolies. Nuclear power requires massive government protection from competition, which makes it attractive to utilities.
Yes, electricity is becoming very cheap in China, and it will be difficult for the West to keep up here. Exorbitant US tariffs are also counterproductive, as they only serve to secure monopoly profits for old utilities.
https://doi.org/10.3390/data10110172
reply