We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows.

It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you.

Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC.

These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts.

The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters.

> When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.

My native language uses some additional CJK chars on plane 2, and before ~2010s a lot of software had glitches beyond the basic plane of unicode. I am forever grateful for the "Gen Z" who pushed for Emojis.

Javascript's String.length is still semantically broken though. Too bad it's part of a unchangeable spec...

I’d be very happy with the standard library providing multiple “length” functions for strings. Generally I want three:

- Length in bytes of the utf-8 encoded form. Eg useful for http’s content-length field.

- Number of Unicode codepoints in the text. This is useful for cursor positions, CRDT work, and some other stuff.

- Number of grapheme clusters in the text when displayed.

These should all be reasonably easy to query. But they’re all different functions. They just so happen to return the same result on (most) ascii text. (I’m not sure how many grapheme clusters \0 or a bell is).

Javascript’s string.length is particularly useless because it isn’t even any of the above methods. It returns the number of bytes needed to encode the string as UTF16, divided by 2. I’ve never wanted to know that. It’s a totally useless measure. Deceptively useless, because it’s right there and it works fine so long as your strings only ever contain ascii. Last I checked, C# and Java strings have the same bug.

The built-in string.length method is useless (it returns the number of char objects) and I agree that's a problem, but the solution is also built into the language, unlike in JS.

The Intl.Segmenter API [0] can similarly yield iterable objects with all the grapheme clusters of a string. Also, the TextEncoder [1] and TextDecoder [2] APIs can be used to convert strings to and from UTF-8 byte arrays.

[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

[1] https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder

[2] https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder

Length in encoded form can be found after encoding by checking the length of the binary content I guess.

I think for historical reasons access to codepoints can be useful, but it's rarely what one wants.

Though you are right in that I don't know of a built-in way to count Unicode Scalar Values (USVs).

The main place I've seen it get annoying is searching for some text in some other text. Unless you normalize the data you're searching through the same way as you normalize your search string.

That’s a great tip - obvious in hindsight but one I’d never considered.

It is valid for the presentation of compound emoji can fallback to their component parts. You can't expect every platform to have an up to date database of every novel combination. A better test is emoji with color modifiers. Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.

On modern desktop OSes and smart phones, I do expect my platform to have an up-to-date unicode database & font set. Certainly for something like the unicode polar bear, which was added in 2020. I'll begrudgingly look the other way for terminals, embedded systems and maybe video games... but generally it should just work everywhere.

Server code generally shouldn't interact with unicode grapheme clusters at all. I'm struggling to think of any common, valid reason to use a unicode character database in 'normal' backend server code.

> Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.

I didn't know about that one. I'll have to try it out.

Case insensitive search

It's fine to have an embedded device that's never updated, but never receives new content - it doesn't matter that a system won't be able to show a new emoji because it doesn't have any content that uses that new emoji.

However, if it is expected to display new and updated content from the internet, then the system itself has to be able to get updated and actually get updated, there's no acceptable excuses for that - if it's going to pull new content, it must also pull new updates for itself.

And if we're talking about how new products should be designed, then the "almost" goes away and they 100% wouldn't receive new content through anything like CDs, the issue transforms from an irrelevant niche (like CDs nowadays) to a nonexistent one.

I despise that Unicode retroactively applied default emoji presentation to existing symbols, breaking old text. Who the hell though that was a good idea?

Before emoji, if somebody open a bug report like: "Your software doesn't handle UTF-8 correctly. It doesn't handle Japanese.",

the response was "Huh? We don't bother to support Japanese. Go pound sand. Close ticket with wontfix.".

Now it's "Your software doesn't handle UTF-8 correctly. It doesn't handle emoji" and we're like "Oh shit! My software can't handle my beloved emoji!"

Google, Apple, IBM and MS also did a lot of localisation so their code bases deal with encoding.

It is FOSS Unix software that had the ASCII mindset, probably as C and C++ string types are ASCII and many programmers want to treat strings as arrays. The MacOS and Windows APIs do take UTF as their input not char * (agreed earlier versions did not put they have provided the UTF encodings for 25 years at least.

You mean like opening a file by name?

Mine is the crying emoji.

And after enough failures in breaking the system, the 100 emoji.

I just tried it in gnome-terminal, and while the crying emoji works fine, polar bear or a country flag causes weird issues.

It's actually worse than that.

Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD.

OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.

What applications? Almost nobody writes their own implementations of encryption nowadays (nor should they). You mean openssl is "broken"?

(Kidding, mostly.)

About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far.

Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare.

Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.

This Joel On Software article [0] is a good starting point. Incredibly it's now over 20 years old so that makes me feel ancient! But still relevant today.

The suggestion that the web should just use utf-8 everywhere is largely true today. But we still have to interact with other software that may not use utf-8 for various legacy reasons - the CSV file example in the original article is a good example. Joel's article also mentions the solution discussed in the original article, i.e. use heuristics to deduce the encoding.

[0] https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you.

For a library, yeah.

https://learn.microsoft.com/en-us/windows/apps/design/global...

[1]Me, surely at least 1 other

[1] Me, for sure.

[1]you may find him at ::1

[2]HEAD~2

[3]Front-end work

If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.

It doesn't. Well, maybe "another".

Your function or even app doesn't need to handle it. Here's what we did on a bookkeeping app: remove all the heuristics, edge cases, broken-csv-handling and validation from all the CVS ingress points.

Add one service that did one thing: receive a CSV, and normalize it. All ingress points could now assume a clean, valid, UTF8 input. It removed thousands of LOCs, hundreds of issues from the backlog. It made debugging easy and it greatly helped the clients even.

At some point, we offered their import runs for download, we added the [original name]_clean.csv our normalized versions. Got praise for that. Clients loved that, as they were often well aware of their internal mess of broken CVSs.

Could this work? Implement handling of ancient excel files in your SaaS product, but charge extra dollar for parsing legacy formats and provide information how to export correct files from excel next time :)

What if you just gave them instructions?

OTOH, they have to follow some process to use the software. For just the CSV export, they already have to ensure column orders, values, formats, maybe headers, etc. Selecting UTF-8 from a dropdown seems like the easy part.

My accrued compatibility shims disagree.

In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused.

You would have lost many of them simply with then jumble of letters “UTF-8”.

Gen Z has skills in what is available and popular today, just like the previous generation had.

With CSV exports, there's already a good bit of training you have to do WRT the column layout, file format, headers (or not), cell value formats, etc. There's far more lift involved in training users there than ensuring they select "UTF-8" when they select CSV.

And, there's really nothing technical about it, as they don't need to understand what "UTF-8" actually means any more than they need to understand what "CSV" stand for. It's a simple ask in a list of asks.

I weigh this against having every developer now believe they have to check/convert every character set, which can be unreliable and produce garbage, BTW. And, speaking of garbage, there are some encodings that you won't be able to convert in any case, so it would be technically impossible to preserve data integrity without pushing some requirement on the source up to the user.

So, it's about tradeoffs. And, again, if asking users to choose UTF-8 is the difference between customers choosing your app or a competitor's, then you probably need to be more worried about that than your charset encoding.

Of course, if that charset uses code points not available in your app's native charset, then you're kind of back to square one (unless your use case tolerates garbled or missing data).

It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.

I assume you mean "rip them", as in transcode to a different format?

In that case, you need a tool that takes the old input format(s) and convert them to the new format.

For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8.

Isn't the point of the article to describe how an engineer would create such a tool?

Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job.

If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.

But the point of the article is not the same thing as how well it achieved its goals.

This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.

In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|".

So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.

It depends on the requirements.

If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.

Yes, there are scenarios where that matters. However the vast majority of usecases of utf-8 don't fit that or even use encryption at all.

And even if the format only stored the checkbox states as a single bit each (unlikely), compression algorithms don't care. They will behave differently on different byte sequences, which can easily lead to a difference in output length.

Also, it's already been done with voice calls with no attacker-controlled data: https://web.archive.org/web/20080901185111/https://technolog...

The version of your checkbox example which is vulnerable without any formal data compression is when the checkbox is encoded in a field that is only included or changes in length if the value isn't the default, common in uncompressed variable-length encodings like JSON.

Also, zip/deflate etc was not designed to eliminate side channel leakage. Some compression schemes obviously (with padding) can mitigate leaks, but it has to be done deliberately

The special concern with compression is when attacker-controlled data is compressed against secret data because then the attacker can measure the length multiple times and deduce the secret based not just on the length but on how the length changes when the secret is constant and the attacker-controlled data varies. This can be mitigated with random padding (makes the attack take many times more iterations because it now requires statistical sampling) or prevented by compressing the sensitive data and attacker-controlled data separately.

For example "FooFoo" will be smaller than "FooBar" since there is a repeated pattern in the first one.

The attacker can look at the file size and make guesses about how repetitive the text is if they know what the uncompressed or normal size is.

This gets more powerful if the attacker can insert some of their own plaintext.

For example if the plaintext is "Foo" and the attacker inserts "Fo" (giving "FooFo") the result will be smaller than if they inserted zq where there is no pattern. By making lots of guesses the attacker can figure out the secret part of the text a little bit at a time just by observing the size of the ciphertext after inserting different guesses.

If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which:

A: ;

B: *&

1. the length of your phone call; and

2. what language you were speaking; oh and

3. half the words you said

(i.e. pwned)

https://web.archive.org/web/20080901185111/https://technolog...

How? Remember, the uncompressed text gets encrypted too.

That article is about voice calls. Totally different topic. Nothing to do with UTF-8.

So if you compressed the string "Bob likes yams" and I could convince you to append a string to it and compress again, then I could see how much the compressed length changed.

If the string I gave you was something already in your data then the string would compress more than it would if the string I gave you was not already in your data - "Bob likes yams and potatoes" will be larger than "Bob likes yams likes Bob".

If the only thing I can see about your data is the length and how it changes under compression - and I can get you to compress that along with data that I hand to you - then eventually I can learn the secret parts of your data.

This is true in both the compressed and non-compressed case. However with compression the size of the plaintext depends on the contents, so the leak of the size can matter more than when not using compression.

Even without compression this can matter sometimes. Imagine compressing "yes" vs "no".

Ah, I see. Naïvely, this seems like a really bad thing for an encryption algorithm to do—is there no way around it? Like, why is encryption different from hashing in this regard?

Part of the problem (afaik) is we lack good math tools to analyze the trade offs of different padding size vs how much extra privacy they provide. This makes it hard to reason about how much padding is "enough".

Another approach is adding a random amount of padding. This can be defeated if you can force the victim to resend messages (which you then average out the size of).

Hashing is different because you don't have to reconstruct the message from the hash. With encryption the recipient needs to decrypt the message eventually and get the original back. However there is no way to transmit (a maximally compressed) message in less space then it takes up.

There are special cases where this doesn't apply e.g. if you have a fixed transmission schedule where you send a sprcific number of bytes on a specific agreed upon schedule.

In an enormous number of real world cases adversaries can end up including attacker-controller input alongside secret data. In that case you can guess at secret data and if you guess correctly, you get smaller compressed output. But even without that, imagine the worst case: a 1TB file that compresses to a handful of bytes. Pretty clearly the overwhelming majority of the text is just duplicate bytes. That’s information which is leaked.

- not having to worry about byte order;

- and cruft like surrogate pairs;

- being able to pass the text through 8 bit string representations and even manipulate it as 8 bit in certain useful ways

Add bitmapped graphics to any text, and it will dominate the size.

Size of text is just noise in the world of streaming video, terabyte drives, fiber to the home ...

What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser."

1: https://en.wikipedia.org/wiki/Robustness_principle

If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.

HTML was a massive success, while at the same time when we tried having a strict spec with XHTML, we failed miserably.

I mean, the purist in me wants strictness but it seems lousyness wins.

The explosion of the web happened in no small part because of how easy it was to write some HTML and get a basic, working webpage out of it. If you nested some tags the wrong way and the browser just put up an error page, rather than doing a (usually) pretty good job figuring out what you actually meant, people would get frustrated faster and not bother with it at all.

But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.

The difficulty in drawing a line in the sand and sticking to the spec, though, is that of user blame. Let's say you implement a spec perfectly -- even if you are the originator of the spec -- and then someone comes along and builds something of their own that writes out files that don't conform to the spec. Your software throws up an error and says "invalid file", but the other piece of software can read it back in just fine. Users don't know or care about specifications; they just know that your software "doesn't work" for the files they have, and the other software does. If you try to tell them that the file is bad, and the other software has a bug, they really won't care.

That's how Perl ended up the way it is.

It also shows the harmfulness of binary black and white thinking in engineering. There are choices other than "just let everyone do whatever and hope all the different things picking up the pieces do it in more or less the same way" and "rigidly specify a spec and blow up the universe at the slightest deviation". Both of those easy-to-specify choices have excessive costs. There is no escape from hard design tasks. XHTML may always have been doomed to fail, but that is not to say that HTML had to be allowed to be as loosey-goosey as it is, either.

Had a gradient of failure been introduced rather than a rigid rock wall, things very likely wouldn't have gotten as badly out of hand as they did. If, for instance, a screwed up table was specified to deliberately render in a very aesthetically unappealing manner, but not crash the entire page the way XHTML did, people would have not come to depend so much on HTML being sloppy. The resulting broken page would still be somewhat usable, but there would have been motivation to fix it, rather than the world we actually live in where it all just seemed to work.

>> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded.

>> If you'd like to see why, read the HTML 5 parsing portion of the spec.

> Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec

I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?

Being lenient is all well and good when the consequences are mild. When the consequences of misinterpreting or interpreting differently to a second implementation becomes costly, such as a security exploit, then the Robustness Principle becomes less obviously a win.

It's important to understand that every implementation will try to fix-up formatting problems in their own way unique to their particular implementation. From that you get various desync or reinterpretation attacks (eg. HTTP request smuggling).

As you say, unfortunately the genie cannot be put back in the bottle for formats that already have defective implementations in the wild.

E.g. you produce valid .wat files, but my software which also outputs those has some bits screwed up.

My program can read both .wat but yours can't, but I have 5% market share.

Your users complain they sometimes receive files your software can't read while the competitor can. Do you tell them "well that file is invalid, tell whoever sent it to you to change the software they use"?

The genie can't stay in the bottle unless you have some sort of certification authority and even that may not be enough (see USB)

And you're saying that by doing this, not only do I gain an advantage over the existing competition, but I also make it more difficult for more competitors to appear?

[takes notes]"

The customer does not know. They just want it to work. They may be using something that someone else gave them. The original source system of the file may not be changeable. But most importantly, their boss just wants it to work. or else.

But that is not always the case, and e.g. silently "fixing" text encoding issues can often corrupt the data if you get it wrong.

By all means offer options of you want, but if you do flag very clearly to the user that they're taking a risk of corrupting the data unless any errors are very apparent and trivial to undo.

This basically loses data integrity if it's wrong though.

You might want to do that with human input if it's helpful to the user - ie user enters a phone number and you strip dashes etc. But if it's machine to machine, it should just follow the spec.

If you reject malformed input, then the person who created it has to go back and fix it and try again. If you interpret malformed input the best you can (and get it right), then everyone else implementing the same thing in the future now also has to implement your heuristics and workarounds. The malformed input effectively becomes a part of the spec.

This is why HTML & CSS are the garbage dump they are today, and why different browsers still don't always display pages exactly alike. The reason HTML5 exists is because people finally just gave up and decided to standardize all the broken behavior that was floating around in the wild. Pre-HTML5, the web was an outright dumpster fire of browser compatibility issues (as opposed to the mere garbage dump we have today).

Anyway, it's not really important to try to convince you that Postel's Law is bad; what's important is that you know that many people are starting to think it's bad, and there's no longer any strong consensus that it was ever a good thing.

Bush hid the facts

For those who don't know that reference: https://en.wikipedia.org/wiki/Bush_hid_the_facts

(A text file containing only the ASCII bytes "Bush hid the facts", when opened in Windows Notepad, displays a sequence of CJK characters instead of the expected English sentence.)

https://en.wikipedia.org/wiki/Robustness_principle#Criticism

Postel's Law doesn't pass a software engineering smell test.

The idea that software should guess and repair bad inputs is deeply flawed. It is a security threat and a source of enshittification.

I am so over this word.

I took the article to be for people who would be writing that "standalone program"?

I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.

Me: "I'll make the program stop and ask the user."

Customer: "No. We have 10 million files. Start by using heuristics for batch processing."

Or "what do you mean ask them, they barely know what a jpeg is?!"

You just look bad if you can't do what every other software is able to do. Charging for it takes that to another level.

That's how business works...

If such a business competitor doesn't exist, then yes charge extra, and actually do the work correctly.

This isn't some theoretical thing, we do this at $dayjob right now not only guessing the encoding but the file-type as well so that we can make sense out of whatever garbage our users upload. Everything from malformed CSV exports form Excel to PDFs that are just JPEGs of scanned documents. It works, and it works well.

And of course it does, the files our users are handing to us work on their machines. They can open them up and look at them in whatever local software they produced them with, there's no excuse for us to be unable to do the same.

The FCC ULS database records are stored in a combination of no fewer than three different encodings(1252, UTF8, and something else for a handful of German names) that vary per record.

When I brought this up they said something to the effect of: it's already unicodes it has tilde letters!

I had write to my own "clean this as well as possible" thing, and it did a good enough job.

Browsers do, kind of https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-...

Browsers used to have a menu option to choose the encoding you wanted to use to decode the page.

In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this.

They seem to be in the process of disabling that option too:

> Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported.

( https://support.mozilla.org/en-US/kb/text-encoding-no-longer... )

This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!

https://hsivonen.fi/no-encoding-menu/

> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run.

> Elaborate UI surface for a niche feature risks the whole feature getting removed

> Telemetry [...] suggested that users aren’t that good at choosing correctly manually.

In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)

>> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden).

There's no explanation of why you'd want this, or why it's security-relevant.

(Farther down, there's a mention of self-XSS, which definitely isn't relevant.)

>> Elaborate UI surface for a niche feature risks the whole feature getting removed

They've already removed the whole feature. That was easier to do after they mostly disabled it, not harder.

>> Telemetry showed users making a selection from the menu when the encoding of the page being overridden had come from a previous selection from the menu.

That would be an example of "working as expected". The removal of the ability to do this is the problem that disabling the encoding menu causes! Under the old, correct approach, you'd guess what the encoding was until you got it right. Under the new approach, the browser guesses for you, and if the first guess is wrong, screw you.

Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8.

If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.

Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.

If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.

Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.

I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.

Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.

Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.

The reason its usually safe is that to exploit you need:

- a secret inside the html file

- the secret has to stay constant and cannot change (since it is adaptive attack. CSRF tokens and similar things usually change on every request so cannot be attacked)

- the attacker has to have a method to inject something into the html file and repeat it for different payloads

- the attacker has to be able to see how many bytes the response is (or some other side channel)

- the attacker is not one of the ends of the communication (no point to attack yourself)

Having all these requirements met is very unlikely.

[0] https://utf8everywhere.org/#asian

(Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)

Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.

> If you give me a computer timestamp without a timezone, I can and will assume it's in UTC.

Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself.

The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken.

My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm.

But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.

Consider my statement more in the context of logs of past events. The only time you can reasonably assume a given file is in a particular non-UTC TZ is when it came from a person sitting in your same city, from data they collected manually, and you're confident that person isn't a time geek who uses UTC for everything. Otherwise there's no other sane default when lacking TZ/offset data. (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).

That's certainly fair in the context of a recurring event with some formula. I caution that a lot of people will still immediately reach for timestamps to calculate that formula, particularly for a next occurrence, and in the context if this conversation they would be given as an ISO8601 datetime based on Local Time. I would also caution that calendar events that have a distinct moment in time that they start are also prime for Local Time where a UTC-default mentality will cause errors.

> Consider my statement more in the context of logs of past events

From the stance of computer generated historical log data, I definitely agree that UTC everywhere is a sane default and safe to assume :)

(And, in your defense, I would definitely argue UTC-everywhere gets you 95% of the way there for 5% of the effort... I get why people make the tradeoff)

> (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).

More nitpicking on my part, again, I'm sorry, it lets you convert from one _offset_ to another or from an offset to UTC. Think Arizona being a special snowflake who (mostly!) doesn't observe DST. You can't assume all UTC-7 offsets are all Mountain Time.

"Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.

People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.

UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need an Order. Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work?

Others variants of Unicode have BOMs, e.g. UTF-16BE. We know CJK languages need UTF-16 for compression. The BOM is only a couple more bytes. No problem, so far so good.

But there are old files, that are in 'platform encoding'. Fine, let there be an OS 'locale', that has a default encoding. That default can be overridden with another OS 'encoding' variable. And that can be overridden by an application arg. And there may be a final default for a specific application, that is only ever used with one language encoding. Then individual files can override all of the above ...

Text file (serialization) formats should define handling of optional BOM followed by an ASCII header that defines the encoding of the body that follows. One can also imagine .3 filetypes that have a unique or default encoding, with values maintained by, say, IANA (like MIME types). XML got this right. JSON and CSV are woefully neglectful, almost to the point of criminal liability.

But in the absence of all of the above, the default-default-default-default-default is UTF-8.

We are talking about the future, not the past. Design for UTF-8 default, and BOMs for other Unicodes. Microsoft should have defined BOMs for Windows-Blah encodings, not for UTF-8!

When the whole world looks to the future, Microsoft will follow. Eventually. Reluctantly.

If you assume UTF-8, you will have corrupted text.

I agree that I'm mad about Excel outputting Win-1252 CSV by default.

If the world worked that way, it would be a very different world than the one we've got.

> You are suggesting that if software developers willfully refuse to implement

it will happen anyway, sooner or later

> will result in lots of corruption

there are lots of corruption anyway, e.g euro sign, some european letters, whole other alphabets like cyrillic one). The most dangerous are subtle ones like german ẞ)

P.S. I don't have office installed to check, but online version exports non cp-1252 chars as "?" . So nice

P.P.S. Apple Numbers offers choice of encoding on export, with UTF-8 default. Google Sheets exports as UTF-8 without choice

Don't interpret user supplied strings at all. Defines max lengths as byte lengths.

Remain agnostic of encoding. Especially in libraries.

It's easier than people think it is thanks to some very clever people's work a long time ago.

I worked at an AI company that was ahead of its time (actually I worked at a few of those) where the data scientists had a special talent for finding Docker images with strange configurations so all the time I'd find out one particular container was running a Hungarian or other wrong charset.

(And that's the problem with Docker... People give up on being in control of their environment and just throw in five different kitchen sinks and it works... Sometimes)

Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G

Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching.

I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap

There is no CSV specification. That does bring opprobrium. RFC4180 is from 2005, long after Unicode and XML, so people should have known better. The absence of a standard points to disagreement, perhaps commercial disagreement, but IETF should be independent, is it not?

That failure to standardize encoding, and other issues (headers, etc.), has wasted an enormous amount of time for many creative hackers, who could have been producing value, rather than banging their head against a vague assembly of pseudo-spec-illusions. Me included, sadly.

The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.)

Making a CSV spec post-hoc solves nothing, as others here have pointed out, because there is already an "installed base" so to speak. If anything it's worse because it might mislead some people into thinking they can write to the spec and handle any CSV file.

The right move, if you want a nicely precise and strict spec, is to admit it's a new thing and give it a new name, maybe CSVS or something like that.

But good luck making it popular - there are plenty of CSV libraries out there that handle most CSV files well enough, just as there is tons of software that handles HTML and RSS well enough (which is why I am a fan of both those formats).

This is a surprisingly deep thought. I'm going to need a couple of days to reflect on this...

Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem.

You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible.

But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now?

[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)

The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is.

I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever.

There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!

So we end up at another plausible future-directed design decision: computer-computer communication should use HTTP. I think many systems have ended up there already, perhaps prompted by the issues we have discussed.

Moral: good specs attract usage; bad, incomprehensible or inconsistently implemented specs fade away.

That’s … not true? Most Chinese software and websites are utf8 by default, and it’s been that way for a while. GBK and her sisters might linger around in legacy formats, but UTF-8 has certainly reached the point of mass adoption from what I can tell.

My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest.

(.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.)

> JSON and CSV are woefully neglectful,

As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1

That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane".

(Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.)

> When the whole world looks to the future, Microsoft will follow.

That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).

My take though is that CSV is not a good thing because the format isn't completely standardized, you just can't count that people did the right thing with escaping, whether a particular column is intended to be handled as strings or numeric values, etc.

Where I work we publish data files in various formats, I'm planning on making a Jupyter notebook to show people how to process our data with Pandas -- one critical point is that I'm going to use one of the commercial statistics data formats (like Stata) because I can load the data right the first time and not look back. (e.g. CSV = good because it is "open" is wrong)

If I am exporting files for Excel users I export an Excel file. Good Excel output libraries have been around for at least 20 years and even if you don't have fun with formulas, formatting and all that it is easy to make a file that people will load right the first time and every time.

I wish that were the case more often. Depends on your ecosystem, of course.

For instance, I've yet to find a good XLSX library for JS that works well "idempotently" (in the browser as well as Node/etc). Every one I tried either had native dependencies and couldn't run in-browser or had a cost (time, money, size) I couldn't budget for at the time.

I have found some XLS libraries for JS that were extremely "mid", but outputting XLS is nearly as bad as CSV in 2024. (Including all the huge messy legacy of character set Encoding problems.)

The best and worst thing about CSV is that it seems "low overhead": it seems really cheap to output. ("How hard can it be, just ','.join(records)?" has been the pit so many of us fall into over and over again and sometimes never come out.) In terms of low overhead: in a world where all my APIs are already talking JSON if I can wrap an existing HTTP API with just two extra headers to get "free" Excel files for my users, that could be a beautiful world:

  Content-Type: application/json+excel
  Content-Disposition: attachment; filename="BusinessUsersDumbReportRequest.xljson"

Secondly for "ownership" reasons: We don't want to give non-developers the mistaken impression that Excel "owns" JSON and that it is a Microsoft format. I've had people tell me that CSV must be a Microsoft format because you can double click them in Excel and they show an Excel-like icon (in some ways it has been too long since Lotus existed and Excel was in the "we'll take all of our competitors' file associations too" era). On the one hand it might be nice to blame all of CSV's problems on Microsoft and Excel if that were actually the case, but on the other hand it also confuses people as to the real/valid uses of the format. Unfortunately, too, that transitive relationship goes both ways and I've heard second hand that CSV files are among the reasons the Excel team hopes to never add another file type association again because supposedly they get far too many support requests for CSV file problems that maybe shouldn't be their job to deal with.

A separate file association adds some intent of "this file was actually meant to be opened in Excel and hopefully the developer actually tested it some".

So that you know you are dealing with UTF-8. Assuming ASCII only works if you are only dealing with English texts and data.

What you do, rather, is drop support for non-UTF-8.

Work with tech-stacks whose text handling is based strictly around Unicode and UTF-8, and find enough opportunities that way that you don't have to care about anything else.

Let the customers who cling to data in weird encodings go to someone who makes it a nice to support that.

> What do web browsers do if they don’t find any Content-Type, either in the http headers or the meta tag? Internet Explorer actually does something quite interesting: it tries to guess, based on the frequency in which various bytes appear in typical text in typical encodings of various languages, what language and encoding was used. Because the various old 8 bit code pages tended to put their national letters in different ranges between 128 and 255, and because every human language has a different characteristic histogram of letter usage, this actually has a chance of working. It’s truly weird, but it does seem to work often enough that naïve web-page writers who never knew they needed a Content-Type header look at their page in a web browser and it looks ok, until one day, they write something that doesn’t exactly conform to the letter-frequency-distribution of their native language, and Internet Explorer decides it’s Korean and displays it thusly, proving, I think, the point that Postel’s Law about being “conservative in what you emit and liberal in what you accept” is quite frankly not a good engineering principle.

1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

Ideally people would see gibberish (or an error message) immediately if they don't provide an encoding; then they'll know something is wrong, figure it out, fix it, and never have the issue again.

But if we're in a situation where we already have lots and lots of text documents that don't have an encoding specified, and we believe it's not feasible to require everyone to fix that, then it's actually pretty amazing that we can often correctly guess the encoding.

If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.

Don't really even need to do that. There's only a handful other encodings still in common use, just try each of them as fallbacks and see which one works without errors, and you'll manage the vast majority of what's not UTF-8.

(We recently did just that for a system that handles unreliable input, I think I remember our fallback only has 3 additional encodings before it gives up and it's been working fine)

Since UTF-8 is a variable-length encoding, it somewhat naturally has some error detection built in. Fixed-length encodings don't really have that, and for some of them, any byte value, 0 to 255, in any position, is valid. (Some have a few byte values that are invalid or reserved, but the point still stands.)

So you could very easily pick a "next most common encoding" after UTF-8 fails, try it, find that it works (that is, no bytes are invalid in that encoding), but it turns out that's still not actually the correct encoding. The statistics-based approach will nearly always yield better results. Even a statistics-based approach that restricts you to a few possible encodings that you know are most likely will do better.