The article doesn’t provide any alternatives?

I think there are indeed many challenges when evaluating Compound AI Systems (http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems...)

But evals in complex systems are the best we have at the moment. It’s a “best-practice” just like all the forms of testing in the “test pyramid” (https://martinfowler.com/articles/practical-test-pyramid.htm...)

Nothing is a silver bullet. Just hard won, ideally automated, integrated quality and verification checks, built deep into the system and SDLC.

Is this a response to Chainguard? https://www.chainguard.dev/

Exciting!

I'm surprised to see nobody has mentioned gradio yet. It seems to sit in the same niche as streamlit and mesop.

https://gradio.app/

+1

Bazel and monorepo are a bit exotic to most enterprise Java devs, but now that I've experienced it, I'll never go back to Gradle or Maven or sbt.

maven can do monorepos pretty easily i thought. The potential benefit of bazel is the better level of cache compiled artifacts that maven doesn't seem to handle very well (and so you almost always end up recompiling everything just in case...)

Shopify and Target have been known to use it.

This is neat!

The nixos:latest reference is amusing. A reproducible image from a non-reproducible reference.

Thanks for publishing this. I quickly skimmed the paper, I saw the impressive linear scaling as you scaled to 16 nodes. How long did it take to train the various models in wall clock time?

Link to PEP 684 https://peps.python.org/pep-0684/

The comments at the rationale section of that pep with a "why are we here" feel are more expressive than I expected.

"It can only end in tears"

Any thoughts how to detect direct DML on the state table? Presumably allowing direct DML on the state table without the same on the Outbox table would lead to silent data corruption or lost updates.

What’s the story behind google3? What happened to google and google2?