Programmable configuration language like dhall an jsonet are very powerful and I like them way more than for example helm charts, where templating happens on the string level instead of at the data level.
Just my two cents but I would go against the first interaction with the user being find the bug, I didn't didn't find it and missed the point, granted I didn't take much time and I'm on mobile, but I guess that's the reality for many people if not most :)
The typo is in the name bill in the value of publicKey. I assume this is a way to introduce the user to definitions that help you avoid these kind of mistakes. I think it would be more suitable as a use case description rather than an exercise for the user.
All the people suggesting to use JS instead of Jsonnet are completely missing the point.
Jsonnet is a functional lazy evaluated language that supports powerful referential concepts. Unfortunately the website sucks at getting this across.
JS on the other hand is an imperative programming language.
If you say "use JS" you may as well say "use Python", the 2 are equivalent in this context and neither are in the same class as Jsonnet in terms of robustness and simplicity.
I didn't get it until I used it but after using it I don't think I would go back to test templating when it comes to dealing with JSON/things that are equivalent. i.e I would use Jsonnet to create JSON to convert to YAML rather than templating YAML directly.
Same goes for generating objects directly in JS/Python/jq - yes it can be done, yes sometimes you can write a nice DSL but in most cases Jsonnet is going to allow you to write said DSL faster using functions and it's going to be simpler and less prone to errors.
I think databricks recently released an alternative jsonnet compiler/renderer written in Scala that's faster than the standard Google one (including a workaround for the slowness of JVM startup/shutdown) but compatible with it: https://databricks.com/blog/2018/10/12/writing-a-faster-json...
It was discussed on HN though I don't have the link handy.
But sure, if you are okay with a single yaml file, that's certainly simpler.
If you are suggesting using Python is same as using JavaScript, you missed the point too.
In this case, JavaScript's syntax is very similar to Jsonnet. Or at least what is shown in the example can be accomplished in a very similar way. (maybe except anchors, which can be replaced by variables)
JS is a lot closer to Scheme and Lisps than an imperative language. In fact, it used to have a lispy syntax before Brenden Eich was forced to change it to make it more approachable.
As a person who has done every exercise in SICP, that argument is total bullshit.
Let's go through the list of things that the author says Javascript doesn't have:
2. Lexical block scope
JS has lexical function scope, so it would trivial to convert a "lexical javascript" to a "function lexical javascript." I think it's unfortunate that JS doesn't have lexical block scope, but it's trivial to implement through nested function definitions, so I don't think it's a real problem.
3. TCO
This isn't a feature of a language, it's the feature of a compiler. It just so happens that the Scheme spec requires the compiler to have TCO. But if the Scheme spec didn't mention this, would it make a Scheme something different?
4. Continuations
Continuations are very rare in Scheme, or any language for that matter. I think someone would be hard-pressed to argue that it's fundamental to Scheme.
6. S-expressions
There are many extensions to JavaScript, (JSGEN in Allegro, and various other libraries in SBCL), that allows one to write Javascript in S-expression syntax. So maybe while Javascript doesn't use S-expressions, one can very trivially transpile S-exp JS to normal JS.
8. Macros (he probably should have said hygenic macros)
It's pretty easy to implement a hygenic macro system on top of Javascript. In fact, people have already done so, one example is sweet.js.
9. Distaste for mutation
I'm not sure that distaste for something can be part of a language spec. But it's a non-argument: if all JS programmers suddenly had a "distaste for mutation" would it be anymore or less like Scheme? I don't think it matters at all. This point is about culture, not language spec.
So, in conclusion, from the above points, one could trivially write a Javascript to Scheme compiler that would have all the features of Scheme, except maybe call/cc. If this is what you define as Scheme, sure, fine, you win: Javascript isn't Scheme. But for the rest of us, it doesn't really matter and JS is damn close to Scheme from a practical perspective.
Moreover, it wouldn't even be that hard to add call/cc to JS! So I don't think that or any of the author's points are very valid.
JavaScript does have lexical block scope since ES6, with the `let` and `const` statements.
And continuations are implemented by convention in nodeJS (through callbacks)
It even is specified as including TCO, a classic scheme feature, but so far it is implemented only in Safari.
Non mutation also seems to be a trend in web development, again thanks to `const`. The only things that make JavaScript not scheme, then, are syntax and the lack of macros. (Which is kind of a result of the more complex syntax)
I have no dog in this fight but saying JS isn't scheme because it isn't a True Scotsman (plus a bunch a appeal to emotion) doesn't make for a compelling counter-argument.
For people who doesn't know, GCL (Generic/Google Config Language) is a language that uses dynamic scope. You can define a `template` object with a variable defined as `external`. This way you can create similar objects by providing a concrete value for those `external`s at the time of instantiating.
The following paper was referenced in Google's borg paper and gives a good overview of the syntax.
Maybe it’s because I’m not a top 1% intelligence that I don’t get this but under what circumstances would configurations become complex enough to where this becomes necessary?
I don't think it's an intelligence thing just what problems one's come across. Generating configs for various deployments where resources change? Generating configs for developer set up? If you start having a lot of environmental variables or private keys it might be more manageable to get them from another secure source.
Let's say you manage a public GCP product. You need to run your job in multiple data center. Naturally the data center name is different between each Job object. You should only define one template and then substitute the data center name in the object.
But wait the product have multiple regions. For each region we will depend on regionalized database etc to improve reliability. We need to have some sort of "mixin" so that certain command line arguments can be consistently applied to all jobs in that region.
And then SWE want to rollout a new feature, as the first step they want to roll it out in one data center first then move to full prod later. Now you need to have a data center level overrides.
Can all of these be done in a generic programming language? Yes sure, but why not use a language that intentionally limit what you can do, is almost side effect free?
IMO this would mostly be useful if you had an architecture that evolved over time where you started with flat files for configuration and later moved to a centralized configuration system, many of which have concepts of references. Duplicating configuration or tapping into some of the advanced functions in here isn’t super helpful when you’re purely dealing with flat files, but this becomes helpful if you’re combining it with some synced external state because you can use this to “smooth over” differences in data shape.
The Lua language was originally a data templateing language [0], and is still very good for that, but like all such things, it grew. Contrary to Greenspun's Tenth Rule [1], Lua's implementation of Lisp is not at all ad-hoc, informally-specified, bug-ridden, or slow.
I guess the only really jsonnet-specific thing here is std.extVar() call, but generally speaking it's just a more flexible JSON file that doesn't cost much more to use (brainpower or physical resources). I use it to mix and match configs while debugging sometimes as you can do things like if statements and adding dicts together.
It's also nice for modularization, you can do things like having a function that returns a dict, so if your config needs two slightly different datareaders you can just pull out all the common parts and call the function twice instead of having two copies of the entire datareader config (with only minor differences).
There's a real need for JSON templating languages not targeting developers.
Tools like Zapier, Customer.io's webhooks, etc. allow integration with arbitrary APIs and have become a standard part of many marketing/sales stacks. These apps use things like Jinja, Mustache, or Liquid which are great for text but not JSON templates.
Curly braces as primary delimiters, not allowing trailing commas on final elements, and other features of JSON make generating it a real hassle.
Jsonnet is too complex to be that tool, but I'm glad people are working in the space.
It's just my gut feeling but for nondevelopers, the config storage doesn't matter as long as there is a good GUI to generate what they need to save or paste into another system.
It's unambigious and can be reasoned through, but it's certainly hard to tell what structure the template is building at a glance. This is not a complex or contrived example.
Syntax highlighting, linters, proper indentation, etc. all help...but when talking about non-programmers who aren't used to this it's an incredible hurdle. People are also lazy and error prone, so you can't rely on indentation to make things clear, especially since it's not always clear how you should indent nesting of the template vs nesting of the json datastructure you're trying to build.
Most modern templating languages have a config option for changing the delimeter. Setting it to something like (( or << might help differentiate things a bit, but then you're throwing away easy use of examples, documentation, stack overflow questions, and so much more that already exists for that language. That documentation and copy-pastable community help is not something to throw away lightly if you're designing a tool for this audience.
It's also worth pointing out that the above template generates subtly incorrect json with trailing commas. In pure mustache it's not possible to get rid of that trailing comma. You have to either change your data (add a "last":true key to the last element of any arrays), or have some kind of post-processing step.
All of the existing solutions are workable, they just aren't ideal.
I hacked together a asdl parser in boost::spirit a while back that used mustache to generate the C++ AST (which was then used to generate its own AST to make sure it all worked) and can attest the curly braces in the templates were a righteous PITA to get right.
If I had to do it all over again I'd probably use the config option and change them to something like '(.', '.)' that doesn't naturally occur in the output language.
Curly braces on US Layout keyboards are fine, they just need to be balanced. On keyboards with German layout curly braces are really hard to type and on Mac keyboards with German layout they are not even marked. So I don’t think any programmer can work with such a layout, yet many people do exactly that.
There's also another competing json templating project from google in the works. Doesn't seem to be production ready yet, but it's open sourced already - https://github.com/cuelang/cue
Some devs from my company just recently published https://jkcfg.github.io/, building on lessons learned from many years of configuring Kubernetes. Since we are discussing JS for config management.
Heads up for your friends, the home page is not very legible on mobile and the menu doesn't work (when you click the icon it disappears and leaves no options).
This is great. I've been looking for something like this and have been considering nix, Starlark and Dhall. This seems to hit a sweet spot between power and familiarity.
One thought I had though is that I wish they didn't use the % operator for string interpolation though and rather the more recent f-string syntax. I wonder whether it will be possible to add that or if there is a reason to prefer one over the other in this context?
You're assuming that all code can be trusted. Jsonnet is much easier to make secure than JS, since it can be statically guaranteed to be safe and side-effect-free (e.g. can't access the file system). This permits evaluating arbitrary untrusted data, which can be a boon to systems like CI servers, databases, or even Kubernetes (think Helm charts).
The script to generate that JSON is code. Jsonnet also generates JSON. So his script is filling the same role as Jsonnet would. The point stands.
He might trust his code, but I was highlighting reasons why this is not true everywhere, and indeed the ability to evaluate untrusted code can open up previously unavailable avenues. For example, storing server-side Jsonnet that can be safely evaluated by a multitenant backend.
Jinja is a text templating language. A valid ninja template may not produce valid JSON/YAML/etc. Sometimes you need more power than a config language gives you, specifically when you want reuse. Helm uses something like jinja, but the template files are often complex and messy (and they are looking into supporting Lua to deal with these problems). CloudFormation built function semantics on top of JSON/YAML twice: once in the form of templates and again in the form of its intrinsic functions. Skycfg uses the starlark language to output YAML, and this is the best approach since it has the right semantics and uses a familiar language (starlark is stripped down Python).
With a deep enough JSON structure with similar enough items, the jq expressions are hard to make or maintain. This is a real world example: jq '.sourceSelectionTrees[].rootNode.children[].children' | jq '.[]? | .action.attributeMapping.attributeSources' | jq -r '.[]? | select(.type == "XYZ") | .dataRef.id'
A very powerful one. You could in theory stream multiple files into jq and have it output effectively arbitrary JSON based on their contents since it supports conditionals.
I don't think it would be very pretty since it's not designed for that use case, but the jq language is certainly powerful enough to allow it.
Jsonnet is much easier to make secure than JS. Untrusted JS needs to be run in a sandbox, can be insecure and contain side effects (e.g. read from the file system). Jsonnet can be statically guaranteed to not just be safe, but side-effect-free.
I needed a templated language recently (couldn’t do with Mustache methods). Looked at jsonnet but it looked quite abandoned. Ended up finding Json-E [0]. Pretty powerful, and give you the option to pass in methods to the context as to expose them. No support for Async methods, but I wrapped around it with some Promises and did the trick.
If your configuration files are so hard to write that you need to use a templating language that is even harder to write, maybe you need to re-think what a configuration is.
Yeah, I guess this doesn't use JavaScript expressions so it wouldn't have the weird stringiness and truthiness rules JS has, which is what I was worried about. Still, there are a lot of other issues with BCL.. I think I would rather see a piccolo style solution done with Skylark instead of BCL+json.
From reading the website I see it actually was inspired by gcl/bcl for kubecfg, so hopefully they used some lessons learned!
Jsonnet has been around for a while, but it looks like stale. Its stdlib is limited and nonstandard and the whole lazy evaluation is something new developers struggle with. I've been using it at work for 2 years and it's sad that it hasn't moved much. I think it lost a huge opportunity if it had more active and more open for innovation development. I'm sorry to say that, but the lead is busy with stuff and is highly opinionated. Also, Jsonnet is slow. We have a few hundred Jsonnet templates and it takes a very long time to compile, which is a pity given it's written in C (now being slowly and painfully rewritten in Go for whatever reasons).
I would be interested to hear what heavy lispers think of this. The whole mixing code and data together feels very lispy, albeit with a heavier syntax here... is this sort of a step off from s-expressions, or does it occupy a fundamentally different place?
I always want to know what the debugging experience is with any new language. Being side-effect free is cool and all but if I can’t actually figure out why it did A and not B then I’m going to get frustrated.
Considering how easy it is to run Javascript anywhere, even small inline statements interpreted dynamically, I can't think of a single reason to use this instead.
https://gocd.io/ allows you store CI/CD pipeline configuration in a git repository.
If you have many different components, you really want to factor out common things into templates (like build, deploy), with the option to disable some stages (some components might not have end-to-end tests, for example), plug in some values etc.
I've long been on the lookout for a template system for data structures for this use case, so I'll take a look at some of the suggestions in this thread.
I have never used a language/tool like this at scale and enjoyed it.
For example, embedding a bash script in a templating language. This is such an incredibly crap idea I don't know how it keeps getting perpetuated. Use of this pattern works as follows: user A writes the script, tests it, shoves it into template. user B modifies the template, commits it, it later breaks silently because it was not tested, because it's in a template language, so you can't lint it and it has no tests written for it (because how did you know it would even get generated based on your input/function, let alone all the places it would be used and how, because templates). user C finds the hidden breakage after tearing out hair, removes the script from template, and writes it as a normal script that takes options and writes a test for good measure. (If you try to use normal dev patterns around the templates, like generate-template-then-commit-to-vcs, relying on and testing the generated versions, your tests have to be reworked whenever you regenerate, and later the generator gradually begins to break, so you just modify the copy in VCS rather than using the template)
A separate problem is that templating languages that give you advanced functionality lead you to want to use that functionality, and the more you do, the harder it is to maintain over time. It may not be readily apparent that trying to make Ansible configs work declaratively requires an annoying amount of complexity, or writing entire applications that cannot be easily reused and calling them "modules" or "plugins". (Whatever you are trying to template/generate, the only way to know for sure you're doing it validly is to have a function that actually parses and outputs the format according to a spec, because just spitting out a template and then externally validating it becomes very difficult to debug, because you don't know why the template is spitting out the output the way it is. So your templating language eventually becomes a compiler)
Since this is a new language, you also have to re-implement everything you normally have for a language, like a plug-in for your IDE, linter, debugger, code analyzer, port the standard libraries to different systems, etc. And of course all new devs have to learn this thing and all its quirks before they can start doing work. An entire ecosystem eventually needs to exist to support its continued use, and if you have problems, you hope this ecosystem continues to flourish so you can get support for this unsupported language tool thing, or you'll be hiring internal devs to do it.
Maybe the problem is these start out as tools for developers, and only when they get used for operations do the anti-patterns surface. I've gone through it several times and it always ends up a PITA.
It's GCL but with a strong type system and termination guarantees.
I personally like the syntax a bit more than jsonnet. It's a bit less noisy.
I'm also a bit biased because I work on https://github.com/dhall-lang/dhall-kubernetes . We're currently working on a 2.0 release so the current state of the repo is a bit in turmoil.
Programmable configuration language like dhall an jsonet are very powerful and I like them way more than for example helm charts, where templating happens on the string level instead of at the data level.