re: Grafana (i'm the author of the linked issue) - i'm quite excited, i do think there's a world of possibilities here.

Two-way sync with a git repo is one possible path, and we've talked a lot internally at GL about how to best support it. My sense is that we can do it with relatively little friction and likely will - but if you're just syncing with a git repo, there's still a lot of arbitrary, opaque repo layout decisions that still have to be made (how do you map a filesystem position for a dashboard to a position in Grafana? In a way that places the dashboards next to the systems they're intended to observe? With many teams? With many Grafana instances?) which induce new kinds of friction at scale.

Fortunately - and not mutually exclusively with the above - by building the system for schema in CUE, we've made a composable thing that we can make into larger systems. That's what we're starting to do with Polly: https://github.com/pollypkg/polly

Conveniently, my parts of a Grafanaconline talk tomorrow discusses both of these https://grafana.com/go/grafanaconline/2021/dashboards-as-cod... :D

This seems really exciting. I haven't had the chance to use Grafana yet; from the linked issue, am I understanding correctly that you'll be able to serialize dashboards to Cue schema, and hence get all the niceties of a structured representation - versioning, non-visual editing, and reproducibility?

I recall seeing another project HN which created dashboards out of a yaml description. This seems like a fantastic idea, given that a lot of business panels and dashboard apps can be implemented with a limited set of UI interactions.

We've been building dashboards in YAML for a while now using Lowdefy - https://github.com/lowdefy/lowdefy

It is very flexible. Will publish more examples soon!

Yep, this was the one! It looks great though I hadn't got around to trying it. Do you think something like Cue makes for a better representation for lowdefy apps than YAML, since it seems to offer better abstraction ability and hence easier to compose?

It's an interesting thought. I'll definitely spend some time to consider how this could work.

Although, writing apps in yaml or json works really well currently. We mostly express logic through Lowdefy operators. We like supporting yaml / json since it is easy to write code to create or update such apps in any language.

> you'll be able to serialize dashboards to Cue schema

It'll be possible to serialize/represent the dashboards in CUE. Here's a handwavy, pseudocode-y example i use in the talk: https://gist.github.com/sdboyer/d76196f94ca78d1e84c739e95e64...

That said, it's not like we're planning on replacing all the "export JSON" buttons in the Grafana UI with "export CUE." One of the interesting properties of defining schemas in CUE is how it allows us to remove schema-defined default values from a dashboard's JSON. The JSON representation can actually look a lot more like a concise CUE representation.

> versioning

Versioning of the Grafana schema is the essential design goal of the "scuemata" system that is under discussion in that epic issue

Versioning of artifacts that are instances of the schema is a key goal with Polly https://docs.google.com/document/d/1GU0DGy-X6z4FVwbJYPsBKRdq...

> non-visual editing

Like, editing something other than raw code in an editor? Yes, this is also something directly enabled by the schema (again, see the Polly doc, the "Produce" heading). For data-intensive tasks, such editing experiences are the only way to see your logic in the context of data, and therefore IMO prerequisite for confidence

> reproducibility

Yup. This already isn't "hard" to do today, but reproducibility gets more complicated at scale - those questions about how to map what's on disk to what's in your Grafana (or whatever app) in my parent comment become more complicated, leading to friction, leading to staleness.

Thank you for the answer! Think I missed the grafana related context, but the design document is really instructive.

It might sound a bit pedantic, but kustomize strictly avoids the "Infrastructure as Code" space and stays in the "Infrastructure as Data" space. The main difference is that since it just deals with "data", you can build any higher level tooling on this. One of the major proponents of this idea is Brian Grant from Google. He tweets about this from time to time. Here is a recent one: https://twitter.com/bgrant0607/status/1404461906186833927

Is this distinction really about whether the customisation language is declarative? It seems to me that Dhall has the advantages Brian Grant attributes to "Infrastructure as Data", although it is an executable specification.

I also think “executable” is more important. Our configs are so large and complex that we need to DRY them up. A type system alone is insufficient.

Thanks for the "why cue" posts. The two key points appear to be inheritance vs. unification and nothing vs. typed. Somehow I'm unable to grok why unification is better than inheritance. Going a bit deeper:

* "Inheritance, is not commutative and idempotent in the general case"

* "A value is always final in CUE, it can only be made more specific."

From an engineering perspective, the latter is definitely more appealing. But I lack well articulated stories to understand how inheritance fails short, and how graph unification fares better. I wonder if there is somewhere a simple concrete example to contrast the not-idempotent inheritance approach vs. the graph unification approach.

I believe they're discussing commutation and idempotency in the sense of types, rather than the sense of values.

Inheritance allows you to override properties/attributes. If you inherit from 2 classes that both specify the same attribute/property, but with different types for the same attribute, one of them takes precedence and overrides the other. A inherits from B inherits from C is not the same as A inherits from C inherits from B if C says attribute X is a string and B says attribute X is an int.

From my understanding, the equivalent graph unification is invalid. If type A is a unification of type B and C, then B and C cannot have any overlap. Each property is either a member of B or a member of C, but never both. It's commutative because A = B | C (A is the unification of B and C) is the same as A = C | B (A is the unification of C and B). If x is a member of B, and I access A.x, I will always end up accessing B. With inheritance, there can be a B.x and a C.x. Which one I end up accessing depends on which one is A's parent.

Inheritance is not idempotent because if A inherits from B inherits from C, then A is implicitly also B and C. However, A can override B's and C's behavior, so I can't trust that calling C.x will always return the same value. It might return the type C has for that attribute, it might return the type B has for that attribute or it might return the type C has for that attribute. You can prevent overriding the types in children, but at that point you've basically built graph unification.

To give a concrete example, Python allows inheritance. If we are provided with this:

    class MyCar:
        # Epoch time for when the car was made
        created_at: int
    
    class MyCarV2(MyCar):
        # Time it was created in RFC3339 format
        created_at: str
    
    class MyCarV3(MyCarV2):
        # Using an actual datetime object
        created_at: datetime.datetime

And we have a function like this:

    def time_since_created(car: MyCar) -> datetime.timedelta:

That function has no idea what the type of car.created_at will be. Mypy will complain at you because it's bad practice, but it's valid inheritance. Even if they all start with same conceptual time, MyCar.created_at, MyCarV2.created_at and MyCarV3.created_at return different types, despite all supposedly being valid instances of MyCar.

Graph unification forces you to pick a single type for each attribute of a single type. Rather than having 3 types that behave differently, graph unification forces you condense them into one:

    class MyCar:
        created_at: typing.Union[int, str, datetime.datetime]

That time_since_created function now knows exactly what type created_at is. Nothing else can change the type of created_at. If you need to add another possible type you have to either add it to the typing.Union, or create a new class. You can't create a subclass of MyCar with a different type for created_at.

Interesting. Vaguely related, I wonder if cuelang has a mechanism to declare disjoint unions, aka. sum types.

cough how can you leave out https://github.com/purpleidea/mgmt/ =D It's in golang, and is the only reactive DSL.

The language is named Go.

This is very interesting! Working through the docs now and I'm enjoying the schema, and I've came up with similar ideas regarding data validation / generation in the past. It's nice to find a project like this! Thanks!

In most projects data validation becomes problematic. In a most of cases the schema could be a lot more defined than what type def offers. This allows for test cases to make sure data fits the model.

We've also been creating a DSL to build web apps. Check out Lowdefy [0] - I'm trying to come up with an "Infrastructure as Code" word for Lowdefy. "UI as config" is the closest fit, but not sure...

[0] - https://github.com/lowdefy/lowdefy