On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
flohofwoe 15 minutes ago [-]
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
Sharlin 24 minutes ago [-]
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
sph 46 minutes ago [-]
> UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
flohofwoe 27 minutes ago [-]
> ...24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
yyyk 12 minutes ago [-]
Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.
Nobody needs more than 4.47 trillion characters. (famous last words)
achille 39 minutes ago [-]
> Ken Thompson: "...i really dont think it is useful. it is like replacing ipv6 with ipv50"
Dwedit 45 minutes ago [-]
FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.
flohofwoe 31 minutes ago [-]
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
sph 42 minutes ago [-]
True, but not all non-UTF8 bytestrings contain 0xFF bytes, so it’s not very useful in practice.
da_chicken 7 minutes ago [-]
Yes, I agree.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
24 minutes ago [-]
mrlonglong 31 minutes ago [-]
I love it.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
flohofwoe 37 minutes ago [-]
Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)
jibal 17 minutes ago [-]
The letter from Ken Thompson is spot on:
"your first 2 extensions (5 and 6 bytes) were clearly envisioned.
the standard (up to 4 bytes) was created to cover the size of
unicode. i thought any more description would be a waste of
paper. i think your extension from 7 to 8 bytes is a little hoaky.
i requires reading the whole string rather than "knowing" the
number of follow on bytes. so, i think the only thing new is the
7 byte version.
i appreciate the mail, but i really dont think it is useful. it is
like replacing ipv6 with ipv50."
snvzz 39 minutes ago [-]
No project is ever safe from complicators.
This is why we need the KISS enforcers.
flohofwoe 23 minutes ago [-]
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
Nobody needs more than 4.47 trillion characters. (famous last words)
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.
i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."
This is why we need the KISS enforcers.
https://xkcd.com/1953/