For what it's worth, MotherDuck and DuckLake will play together very nicely. You will be able to have your MotherDuck data stored in DuckLake, improving scalability, concurrency, and consistency while also giving access to the underlying data to third-party tools. We've been working on this for the last couple of months, and will share more soon.

The article mentioned that DuckDB keeps improving very quickly. The next couple of months of DuckDB are all about stabilization, with no new features getting added. Once it is robust enough it will be declared "1.0". My guess is that will be in late April.

You mentioned OOMs, this has been a focus for a while and ha gotten steadily better over the past few releases. 0.9 added spill to disk to prevent most OOMs. And 0.10, released a couple of weeks ago, fixes a bunch more memory usage problems. The storage format, which another commenter brought up, is now fully backwards compatible.

I'd suggest giving it another try, especially once 1.0 comes out.

It might be getting better, but the examples are currently so egregious that it's tough to keep giving DuckDB a chance.

Example of a query that should never, ever, out-of-memory, but absolutely will in the latest DuckDB:

  COPY
    (
      SELECT
        rs.my_int,
        rs.my_bigint
      FROM
        READ_PARQUET('s3://some/folder/my-large-files-*.parquet')
        AS rs
    )
  TO
    '/my/home/folder/my-large-file.parquet'
    (
      FORMAT PARQUET,
      ROW_GROUP_SIZE 100000,
      COMPRESSION 'ZSTD'
    )
  ;

This query should simply read the two column series selected based on the parquet metadata and then stream the data to the disk.

And yet it will try to load data in memory before crashing.

Does it fail on nightly?

There were some recent fixes: https://github.com/duckdb/duckdb/issues/10737

I've been testing duckdb's ability to scan multi-tb parquet datasets in S3. I have to say that i've been pretty impressed with it. I've done some pretty hairy SQL (window functions, multi-table joins, etc).. stuff that takes less time in Athena, but not by that much. Coupled with its ability to pull and join that data with information in RDB's like mysql make it a really compelling tool. Strangely, the least performant operations were the mysql look ups (had to set SET GLOBAL mysql_experimental_filter_pushdown=true;). Anyway.. definitely worth another look.. i'm using v 9.2

We're being a bit hand-wavy with the offering while we're in "build" mode, because we don't want to sell vaporware. DuckDB is easy to use out of the box, but so is Postgres, and there are plenty of folks building interesting cloud services using Postgres, from Aurora to Neon. And as many people will point out, DuckDB is not a data platform on its own.

For a preview of what we're doing, on the technical side, a couple of our engineers gave a talk at DuckCon last week in Brussels, it is on youtube here: https://www.youtube.com/watch?v=tNNaG7e8_n8

(for context I'm the author of this blog post and co-founder of MotherDuck)