hey all, another perspective that I have been thinking about is if semantic layers are like ORM for but BI dashboards. Actually, they I think its more than BI dashboards since a similar idea applies to Features. Features in ML land are nothing but a Measure + Entity metadata + TTL. So, really its about higher-order semantics and as we move up the stack, we need richer expression to describe our world.

Feature stores explored here: https://www.xorq.dev/blog/featurestore-to-featurehouse

I think my key takeaway building this is that we need better expression systems and Ibis is a great foundation to build yours..maybe you want to build a language for some other domain etc.

PS: I am one of the authors of bsl and co-founder of Xorq.

I am one of the authors of bsl and founder of xorq.

I’m familiar with Xorq. One of features of the Xorq library that I find interesting is that it catalogs data processing (compute) expressions as it compiles, along with call lineage. Makes reuse easier for SQL and non-SQL processing.

FYI - https://github.com/xorq-labs/xorq

As opposed to Substrait? I think there is certainly a path to converting to Substrait as one of the outputs for on-the-wire format to send query plans to various engines. YAML as such gives a human-diffable representation that can be checked-in to git. Let me know your thoughts...we are super early so happy to take in feedback...

this is really useful. In our case, we are inspired by a terminal game called Zork, which is an Activision trademark. So, we tried to be creative and replacing Z with X made sense at the time. Of course, we did not want to be like any "dork", hence the Q.

Now I read this tought process above, I think this leaves much to be desired.

great point...honestly this will be our 3rd rebrand, but worth considering. I am just hoping that we are in a different enough space that it could slide....

Yeah, please don't stick with Xorq, it's so visually similar to Xorg, it looks like the bottom of the g didn't render, which is totally possible on the web.

I think so too...seriously considering changing the name

And how do you pronounce it? EX-ork, Zork, Ex or Que?

pronounced "zork" per: https://www.xorq.dev/blog/introducing-xorq

The point was it's not obvious.

While we considered Narwhals (It's pretty new) over Ibis, but ultimately we decided to go with Ibis since it supports the SQL backends that we needed.

We have ambitions of supporting alternative APIs like Narwhals in future though, that can map polars API to Ibis's internal representation.

We found Ibis to be super extensible. In xorq, we also support pandas UDFs, so if you know pandas you should be pretty well covered. UDFs are pretty nice way to extend the API.

What sort of operations did you find Ibis was missing when you tried? What about it isn't extensible?

this is an example of an out-of-core processing: https://www.xorq.dev/posts/trino-duckdb-asof-join

Anecdotally, TPC-H 10 TB is pretty doable now a days with DuckDB, so xorq goes as far as your engine may take you...

Looks v. cool, how does gizmo handle concurrency for multiple writes? are there any limits that you have observed?

Hey mousematrix, I've tested with 3 concurrent sessions all writing to the same table - and with "BEGIN TRANSACTION" for each.

I opened a TPC-H SF1GB database with GizmoSQL, started 3 concurrent client sessions, and had each session run:

BEGIN TRANSACTION; INSERT INTO lineitem SELECT * FROM lineitem; COMMIT;

None of the 3 sessions were blocked - and all were able to insert into the table, and read the results of their inserts before committing.

It was very cool to see also that writes do NOT block reads from other sessions - the other sessions just could not see uncommitted data.

After committing in one session, the other sessions could see the data that was inserted.

I haven't stress tested the system to get limits, but I thought this little test would prove useful information for your question.

I'll work to do some benchmarking on concurrent inserts, etc.

Updates to the same data by multiple sessions behave a little differently - see details from page: https://duckdb.org/docs/stable/connect/concurrency.html

Thanks for your question!