Oct 17, 2026, 7:08 PM- Oct 17, 2026, 7:08 PM
Replies
Oct 17, 2026, 7:11 PMAs someone who has written more text editors than the average bear we keep recreating this problem. A Unicode code point isn’t one character because you have combining codepoints, whatever abomination is going on with emoji, etc.
You could do something obscene where you use int32’s to store precomposed units and leak into the PUA/>0x10FFFF range for precomposed emoji or whatever but good god what are we doing.
- Oct 17, 2026, 7:27 PM
@rk @datarama i think there's a need to distinguish between computation formats and serialisation formats
for serialisation, Unicode is probably the best compromise we have. it still sucks, but not as much as anything else.
for computation, then as you say, an internal index of precomputed units, mapped to and from Unicode by the OS - all userspace sees is a string of units, each of which is guaranteed to be unique and distinct
for things that have to deal with Unicode code points, as opposed to linguistic units, uint32 leaves a flat 21-bit (thank you, Unicode, for declaring this fixed for all time!) representation. and 11 spare bits to indicate exactly how this letter is connected to the letters ahead of and behind it (such that a letter with all of those 11 bits clear is defined as the end of a combined set) according to the rules pertaining to that code point. but everything is very strongly encouraged to only deal in words (send a word to the text renderer, get a rectangle with that word in it and a padding margin for whitespace back)
if every language has its own rendering library, then emoji can be treated as just another language, which at least keeps the explosion contained
- Oct 17, 2026, 7:30 PM
Ya know. We could do the UTF-8 trick. The number of initial one bits says how many following code points are part of this unit. I don’t think we’d ever really need all 11 bits.
Still requires an O(n) walk though. I was thinking a preprocessing stage where you gather all the composed characters and give them unique IDs in the extra space.