// HACKER NEWS — CYBERSECURITY
UTF-8000: Unlimited UTF-8
No special cases introduced. All properties preserved.
Try out the reference implementation with $ pipx install UTF-8000.
UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
This is a fun standalone project / proposal.
There is nothing special-case-y about the example 22-byte code unit here. It is just a good prototypical example, demonstrating the power of UTF-8000 with multiple start bytes.
There are only two special cases, both of which are inherited from UTF-8: ASCII as is, and 2-byte UTF-8 having 4 mandatory content bits to check against overlong encoding as opposed to 5 for all longer length code units.
Here is anatomical diagram of the example 22-byte code unit from the tldr.
See the glossary for more information on the definitions of the terms.
Byte number four is exciting! It is a continuation byte, a start byte, the final start byte, has content bits, and has only some of the mandatory content bits, which are straddled across the final start byte and first non-start byte.
The main contribution of UTF-8000's specification is clarity on splitting the highest bits of the first byte of UTF-8 code units into self-synchronization bits and start bits, and then making it clear how to stripe the start bits across the continuation bytes if needed, to achieve arbitrarily large code units.
These terms are ordered somewhat by chronology of first requirement, rather than alphabetically, for convenience.