> what should be used instead to encode structured data, or perhaps more importantly what should have been used to encode graph like things such as map/lists/objects in the 2000’s
I think you mean s-expressions instead of lisp and if you do, I'm with you. They are really neat and underused. Now, if you want to carry a whole language runtime for the structured data format you'll probably end up with the "which lisp" kind of problem in which each app end up using something different and then the structured data files are no longer exactly portable (until you devise a portable standard yadayadayada)
I have to admit, once you've seen S-expressions, XML looks insanely bloated and verbose.
The advantage XML would have in that situation is that, because it's so verbose, if you get a malformed XML, you can eyeball-parse it and often figure out how to hand-edit it to make it valid. If it is valid, you can also see exactly how the schema you were sent differs from the schema you expected. An S-expression, having less redundancy, also is potentially more brittle.
If malforming is regular (like someone printf’d or typed in bad xmls), then the same is true for any human-readable format, since you know data and what it should look like. If not, (like in randomly broken packet), then you solve the problem at the wrong level. By-hand error-correcting verbosity is hardly a selling point of the application level protocol.
> By-hand error-correcting verbosity is hardly a selling point of the application level protocol.
It's a selling point while you're trying to figure out how to get it working. Once you have it working, it's not - but by then, you have it working, so why change it?
> Abstract Syntax Notation One (ASN.1) is a standard interface description language for defining data structures that can be serialized and deserialized in a cross-platform way. It is broadly used in telecommunications and computer networking, and especially in cryptography.
> ASN.1 is a joint standard of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and ISO/IEC, originally defined in 1984...
I think ASN.1 was designed for a world with (a) waterfall development and (b) static deployments - ie no over the air updates. Under the circumstances, messing up is simply not an option - hence defining the standard so clearly and for so many use cases.
Today, of course, we treat the entirety of our deployed infrastructure as 'merely' a platform to write code. And not only are experimentation and failure OK, they're positively encouraged. Velocity became important.
You can meaningfully decode any DER encoded ASN.1 structure and serialize it back without any knowledge of the schema. Somewhat surprisingly you cannot do that with all instances of XML documents.
The first thing wrong is that you have to serialize and deserialize it. Operationally, that's inconvenient, and it shows that they're optimizing for network bandwidth. But these days, squeezing the most out of each bit is, in most cases, not a defensible design decision.
Then, once you deserialize it, it's still a printable version of ASN.1. Sure, it's unambiguous, rigidly defined, and standardized. It's still gouge-your-eyeballs-out horrible to try to do anything with.
Say you get an XML message over the wire with a bit flipped. If you look at it, you have a good chance to be able to figure out what went wrong, edit one character, and you can now process it. If you get an ASN.1 message in the same condition, it's pretty much game over (though there may be special tools that could save you).
Say you get an XML, and you don't know the schema. You look at it, and you can see what's going on. You get an ASN.1 where you don't know the schema, and you can be totally sunk. (If I recall correctly, in ASN.1, you can have schemas that are private, that is, not specified in the standard.)
XML (and JSON) have the advantage of names, which makes it slightly easier when it comes to querying (and indeed building indexes) over lots of data. I'd be amazed if there wasn't tons of work on this for S-expressions, but I can imagine it's slightly clunky.
What do you mean by names here? S-expressions have symbols, which serve the exact same purpose as what I think you might mean: an interned string value which can be cheaply used more than once.
Not saying you can't specify some schema here, but there's nothing native to S-expressions that makes it quite as transparent and simple to specify a path into a data structure.
All you need on .NET/Java in 2007 then is a lisp parser/interpreter. I know they are easy to write, but they aren't as easy to write as something you don't have to write at all.
Let's not forget, many one of these configuration files and data formats are one-off hacks that were meant to be replaced by a real format, a real parser, a real DSL etc.
The reason the xml config/dsl/format stuck is because it worked. And it was cheap and easy.
S-expressions have been around since the 1960s. McCarthy was already proposing S-expressions for what XML would become, in 1975. Stop whining and finding excuses, and start using s-expressions.
I'm not trying to pick the best format for the job, I'm usually trying to pick the least bad one that's available IN the platform/standard library I'm using.
Lisp?