I remember that I got a home assignment in an interview for a PHP job. The person evaluating my code said I should not have used UTF8, which causes "compatibility problems". At the time, I didn't know better, and I answered that no, it was explicitly created to solve compatibility problems, and that they just didn't understand how to deal with encoding properly.

Needing-less to say, I didn't get the job :)

Same with Python 2 code. So many people, when migrating to Python 3, suddenly though python 3 encoding management was broken, since it was raising so many UnicodeDecodingError.

Only much later people realize the huge number of programs that couldn't deal with non ASCII characters in file paths, html attributes or user names, because they just implicitly assume ASCII. "My code used to work fine", they said. But it worked fine on their machine, set to an english locale, tested only using ascii plain text files on their ascii named directories with their ascii last name.

(It doesn't even need to be a very good or strongly-enforced type system - Go makes it dangerously easy to convert between `[]byte` and `string` by other-type-system standards, and yet everything works pretty well. It's enough to hitch your thinking and make you realize you need another step.)

It's about the runtime error handling making it mandatory to deal with the error and won't let you panic at run time for this specific error.

It's not the type system, but many dynamic languages are interpreted instead of compiled.

This is quite literally the job of a type system: to impose a semantic interpretation on sequences of "raw bits" and let you specify legal (and only legal) operations in terms of the semantic interpretation rather than the bits.

(And it had nothing to do with UTF-8. Actually, it's at least partially caused by the CPython developers avoiding UTF-8 for poor reasons.)

But even today, codepoint indexing is too easy and doesn't crash so lots of code is still subtly wrong. Memory usage of a string grows 3x if you add a single emoji. Most libraries are still agnostic about whether they take bytes or str so you still get exceptions thrown with no easy solution. (The growing popularity of type hints is fixing this, but that's not really related to Python 3.)

I was delighted recently to stumble on a history of modern women course HIST1158 named Liberté Egalité Beyoncé and I immediately thought two things: 1. Why are our Computer Science courses given unimaginative names? and 2. What a useful test input, I bet some of our systems don't work correctly for this input even though an acute accent is hardly a bleeding edge feature.

I haven't been able to interest any Computer Science professors in fun names for their courses, but I was able in my test environment to name a COMP series course "Untitled Course Name" with a description explaining that "It is a lovely day in the village and there are only two hard problems in Computer Science".

You mean cache invalidation and naming things? :)

Edit: The proposal for anyone interested https://github.com/dotnet/runtime/issues/933

The output of files on disk can be UTF-8. The continued use of UCS-2 (later revised to UTF16) is happening in the runtime because things like the Win32 API which C# uses is UCS-2. The internal raw memory of layout of strings in Win32 is UCS-2.

*EDIT to add correction

[0] https://simonsapin.github.io/wtf-8/

The biggest problem is when you're working in an ecosystem that uses a different encoding and you're forced to convert back and forth constantly.

I like the way Python 3 does it - every string is Unicode, and you don't know or care what encoding it is using internally in memory. It's only when you read or write to a file that you need to care about encoding, and the default has slowly been converging on UTF-8.

There’s 3 valid (and useful) ways to measure a string depending on context:

- Number of Unicode characters (useful in collaborative editing)

- Byte length when encoded (these days usually in utf8)

- and the number of rendered grapheme clusters

All of these measures are identical in ASCII text - which is an endless source of bugs.

Sadly these languages give you a deceptively useless .length property and make you go fishing when you want to make your code correct.

This is also rarely useful unless you are working with a monospace font where all grapheme clusters have the same width, which is probably none if you support double-width characters. More likely what you are interested in is the display length with a particular font or column count with a monospace font.

I would give it to Java outright if not for the fact that C's char type doesn't define how big it is at all, nor whether it is signed. In practice it's probably a byte, but you aren't actually promised that, and even if it is a byte you aren't promised whether this byte is treated as signed or unsigned, that's implementation dependant. Completely useless.

For years I thought char was just pointless, and even today I would still say that a high level language like Java (or Javascript) should not offer a "char" type because the problems you're solving with these languages are so unlikely to make effective use of such a type as to make it far from essential. Just have a string type, and provide methods acting on strings, forget "char". But Rust did show me that a strongly typed systems language might actually have some use for a distinct type here (Rust's char really does only hold the 21-bit Unicode Scalar Values, you can't put arbitrary 32-bit values in it, nor UTF-16's surrogate code points) so I'll give it that.

C’s character type, FWIW, has a use: it more or less indicates the granularity that is efficiently addressable by the host architecture. Trying to use it for more than that is generally not that fruitful, but it definitely has a purpose and it’s pretty good at that.

Finally, speaking of unfortunate decisions, Rust happens to make one that I don’t particularly like: it lets you misalign characters (and panics), which is…not great. It would be much nicer if the view just don’t let you do this unless you specifically asked for bytes or something.

And again, I don't think Java probably needed 'char' at all, it's the sort of low-level implementation detail Java has been trying to escape from so this is a needless self-inflicted wound. I think there's a char in Java for the reason it has both increment operators - C does it and Java wants to look like C so as not to scare the C programmers.

C's unsigned char could just be named "byte" and if signed char must exist, call that "signed byte". The old C standard actually pretends these are characters, which of course today they clearly aren't which is why this is a thread about UTF-8. I don't have any objection to a byte type especially in a low-level language.

Presumably your Rust annoyance is related to things like String::insert? But I don't understand how this problem arises, if you are inserting characters at random positions in a String, that's just going to be nonsense. I can't conceive of a situation where I want to insert characters (or sub-strings) unless I know where they're supposed to go exactly relative to what is in the string already, whereupon it won't panic.

Compare for example decryption, where we learned not to provide decrypt(someBytes) and checkIntegrity(someBytes) even though that's what People often want, it's a bad idea. Instead we provide decrypt(wholeBlock) and you can't call it until you've got a whole block we can do integrity checks on, it fails without releasing bogus plaintext if the block was tampered with. An entire class of stupid bugs becomes impossible.

Java should have provided APIs that work on Strings, and said if you think you care about the things Strings are made up of, either you need a suitable third party API (e.g. text rendering, spelling) or you want bytes because that's how Strings are encoded for transmission over the network or storage on disk. You don't want to treat the string as a series of "characters" because they aren't.

The idea that a String is just a vector of characters is wrong, that's not what it is at all. A very low level language like C, C++ or Rust can be excused for exposing something like that, because it's necessary to the low-level machinery, but almost nobody should be programming that layer.

Imagine if Java insisted on acting as though your Java references were numbers and that it could make sense to add them together. Sure in fact they are pointers, and the pointer is an integral type and so you could mechanically add them together, but that's nonsense, you would never write code that needs to do this in Java.

K&R C claimed that char isn't just for representing "ASCII" (which wasn't at that time set in stone as the encoding you'll be using) but for representing the characters on the system you're programming regardless of whether they're ASCII. 'A' wasn't defined as 65 but as whatever the code happens to be for A on your computer. Presumably the current ISO C doesn't make the same foolish claim.

These requirements are perfectly good for the needs of a CHARacter type. If you need to control signed / unsigned because you want to use the char as a small integer, you can specify yourself whether it is signed or not.

In reality, where chars are used to store ASCII, the signdness of the datatype is meaningless because the highest bit is never set.

It also wastes valuable code space that could be used to have more characters encode as 3 bytes instead of 4 in UTF-8.

For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8.

But when performance isn't critical, utf-8 should be the default. I don't see a reason for any other encoding.

If you're hoping that a fixed offset gives you a user-percieved character boundary, then you're not handling composed characters or zero-width-joiners or any number of other things that may cause a grapheme cluster to be composed of multiple UTF code points.

The "fixed" size of code points in encodings like UTF-32 are just that: code points. Whether a code point corresponds with anything useful, like the boundary of a visible character, will always require linear-time indexing of the string, in any encoding.

(*) Approximately nothing. If you're in a position where you've somehow already vetted that the text is of a subset of human languages where you're guaranteed to never have grapheme clusters that occupy more than a single code point, then you maybe have a use case for this, but I'd argue you really just have a bunch of bugs waiting to happen.

Source code manipulation is frequently Unicode aware but doesn't care about combinations or things outside of a strict subset of Unicode to modify lexing control flow.

Being able to store (and later refer to) character offsets in the source code is a plus because they'll only ever occur in places where the strict subset is enforced.

This is especially true of languages with line-only comments, etc, where different writing systems being used won't affect the error message information.

Like I said, there are a few useful cases where having a fixed width encoding is beneficial. It's less helpful to the discussion to assert you know better for every case, ever.

It doesn't add anything over storing byte offsets into an UTF-8 string.

What about UTF-256? Maybe not today, maybe not tomorrow, but someday...

E.g., Ruby runtimes scan the bytes in a string and then cache data about them in value called a code range. Knowing the code range, you can optimize many operations to not require additional linear scans of the string. Knowing a UTF-8 string consists only of ASCII characters can allow operations to be just as fast as if the string truly were ASCII-only (Ruby supports 100+ string encodings). And that fact is used throughout the core library to provide fast implementations of many operations (upcase, downcase, capitalize, gsub, substring, and so on). Moreover, a JIT can generate extremely tight code in those situations. Having to take a linear pass through the string to discover codepoint boundaries incurs a huge performance cost. While all strings could be treated uniformly and use Unicode tables for case mapping and such, the extra overhead is brutal. It has a measurable impact on string-heavy applications, such as template rendering and text processing.

In the most general case, yes, you know nothing about the string and can't make any assumptions. You can't even be sure the byte sequence is valid UTF-8. But, very often you do know properties of those strings. And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file).

With the exception of conversion from numbers (which has its own optimizations that are likely equally applicable in UTF8 since Arabic numbers are just ASCII anyway), I’d say all of your examples sound like bugs waiting to happen.

Why shouldn’t string literals be allowed to contain complex emoji? Why should identifiers disallow them? Why should there be a company policy around putting complex emoji places?

Just saying “let’s just declare things such that strings aren’t allowed to have multi-code point grapheme clusters” sounds great until you accidentally let that assumption leak into a place where a user wants to use an emoji and can’t make it match their skin tone.

I’d also say that such restrictions are putting the cart before the horse; the typical reasons for restricting the allowed character set, are precisely because you want to make lazy assumptions about things like string offsets. Saying that such assumptions are a good thing because you have these restrictions in place, seems like circular logic to me.

I didn't say they shouldn't, just that many do not and you know that at parse time.

> Why should identifiers disallow them?

I don't write the language specs. Many languages don't allow classes, methods, variables, etc. to have complex grapheme clusters in them.

> Why should there be a company policy around putting complex emoji places?

Performance. Code sanity. Indexing. Ease of typing. Again, I'm not the one writing the policies. But, they exist.

> Just saying “let’s just declare things such that strings aren’t allowed to have multi-code point grapheme clusters” sounds great until you accidentally let that assumption leak into a place where a user wants to use an emoji and can’t make it match their skin tone.

I'm making a clear distinction between situations where you have user-supplied data and data under control of the language runtime, developer-created files, or those just adhering well-defined file formats. These are all strings and commonly consist of simple codepoints; indeed, many times they're just ASCII characters.

I addressed user-supplied values when I wrote "And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file)." TruffleRuby, for example, uses ropes as its underlying structure, so if you have a template written using all ASCII characters (rather common) and interpolate a user-supplied value, you can put the user string in one rope, the template in others and link them all together into a tree with ConcatRopes. The template ropes still know they only have simple codepoints and operations on those parts can be fast. The user variable only knows it's a generic UTF-8 string and operations on that string, if any, can go down the slower path. Oftentimes, there are no operations to perform on that user string other than to display it. Its mere presence doesn't need to adversely affect the rest of the template.

> I’d also say that such restrictions are putting the cart before the horse; the typical reasons for restricting the allowed character set, are precisely because you want to make lazy assumptions about things like string offsets. Saying that such assumptions are a good thing because you have these restrictions in place, seems like circular logic to me.

I'm not making any assumptions. I've spent an awful lot of time optimizing string performance in the context of a Ruby runtime and your initial claim of constant-time subscripting being a myth doesn't match my experience. Ruby allows as complex of a string as you want, but the reality is there are many situations where strings, by either by restrictions or de facto, will not have multi-codepoint grapheme clusters. In many situations you'll have strings with all the codepoints in the ASCII range. If the only information the runtime records when parsing a string is that "this is a UTF-8" string and then operates on all UTF-8 strings uniformly, you leave a lot of performance on the table. The best performing situation is when you don't have to deal with variable-width codepoints in a UTF-8 string. UTF-16 and UTF-32 aren't terribly common in Ruby, but they exist as valid encodings (well, UTF-16BE/UTF-16LE and UTF-32BE/UTF-32LE) and have simpler execution paths than UTF-8 for many use cases.

Assuming you mean different encoding forms of Unicode (rather than entirely different and far less comprehensive character sets, such as ASCII or Latin-1), there are very few use cases where "subscripting" or "length calculations" would benefit significantly from using a different encoding form, because it is rare that individual Unicode code points are the most appropriate units to work with.

(If you're happy to sacrifice support for most of the world's writing systems in favour of raw performance for a limited subset of scripts and text operations, that's different.)