Oct 4, 2026, 2:51 AM@gregeganSF This is interesting because to my understanding of how "transformer models" work, you'd expect them to sometimes get confused and meander in a different direction mid-word, because I was told stems and suffixes of words get split into separate tokens. And I've been going, but I've never seen that happen in the wild, so I must misunderstand something. But I *think* you can see it happening in your email there.
Oct 4, 2026, 2:55 AM@gregeganSF Anyway, I think "Morphological Regency", if it existed, would probably be written by… Susanna Clarke, maybe? It sorta almost works as a not-as-good alternate title for Piranesi…
@mcc @gregeganSF pure speculation, but some kind of shitty check that the words are recognized and if it detects non-words it just tries again would be on-brand.
@mcc @gregeganSF each output is a probability distribution for all tokens. the concept of temperature scales & saturates the probability. then the sampler picks from tokens in e.g. the top 95% percentile (min_p), then limits that set to e.g. the 20 most likely (min_k), and then does a weighted random pick from that set.
for ambiguous junctions, the distribution is flat rather than S-curve shaped, and odd choices are more likely.
when temperature is high, oddness is very likely.
@mcc @gregeganSF I guess common large LLMs are too large, and the prior from the word beginning is too strong for it to happen with any regularly, but I have seen Frankenwords and misspellings occasionally when using smaller, local models. The spammers probably use the smallest model they can (mostly) get away with.