Oct 9, 2026, 7:48 PM- Oct 9, 2026, 7:48 PM
Replies
Oct 9, 2026, 8:17 PM@cwebber @gaditb @bremner @aparrish what does "strings are unicode" mean? "canonically equivalent strings measure as equal"? or "strings are sequences of grapheme clusters with the structure, but not semantics, of unicode extended grapheme clusters"? or just "strings are sequences of code points in the range 0x0000 to 0x10FFFF with code points in the range 0xD800 to 0xDFFF forbidden"? or "strings are sequences of code points in the range 0x0000 to 0x10FFFF with code points in the range 0xD800 to 0xDFFF allowed"?
Oct 9, 2026, 8:31 PM@u2764 @cwebber @bremner @aparrish I assume "string and bytesequence are different types", and in particular "slicing and joining and length and nth-character selection and char-by-char iteration and regular expressions and ..., transform valid unicode sequences to valid unicode sequences, and don't split or choke on multi-byte single-codepoint sequences".
- Oct 9, 2026, 8:36 PM
- Oct 9, 2026, 8:42 PM
@gaditb @cwebber @bremner @aparrish but most people use “supports unicode” to mean “has an underlying data type which can support numbers up to 0x10FFFF”, which does not really involve Unicode at all, since Unicode does not mandate that characters have a particular byte representation
what is a “multi-byte sequence” in this context? 好 and æ̇ are both characters which take up multiple bytes, but one is a single codepoint and one is two. do you treat these the same or differently?
- Oct 9, 2026, 8:47 PM
@gaditb @cwebber @bremner @aparrish “naïve” and “naïve” are canonically equivalent strings. if a program expects one and a user inputs the other, this should be treated as a match. the lengths of these strings should be the same, and iterating their characters should produce canonically equivalent grapheme clusters. that is what the unicode standard specifies behaviour should be
- Oct 9, 2026, 8:50 PM
allisonaparrish@friend.campOct 9, 2026, 9:07 PM@u2764 @gaditb @cwebber @bremner yeah, really what i mean by "strings are unicode" is that i don't have to profusely apologize to my students who want their programs to work with languages that don't easily fit into the one-byte-per-character paradigm. python strings are sequences of unicode code points, which is at least a good starting point for this, especially w/the unicodedata library (for canonicalization and grapheme cluster iteration). something like that would be an okay baseline imo
- Oct 9, 2026, 8:35 PM
@cwebber @gaditb @bremner @aparrish (i am dubious that you can have both “a small team could reasonably re-implement the language from the standard” and “strings are unicode”. a small team CANNOT reasonably re·implement the unicode standard. language is complex and requires a team of specialists to properly handle. you can outsource this work to ICU, but that means either shipping ICU in your binaries or giving up on easy crossplatform compatibility. and that would still be “a small team could easily re·implement except this one core part”. sorry)
- allisonaparrish@friend.campOct 9, 2026, 10:54 PM