@jasongorman One of the major eye-openers for me when it comes to human cognition was when reading Oliver Sack's fantastic "The man who mistook his wife for a hat" and the research done on split-brain patients.

It turns out that not only are you correct about LLMs, but humans do exactly the same. We will happily make up whatever plausable story explains our behavior after the fact - fully believing it to be true ourselves.

https://en.wikipedia.org/wiki/The_Man_Who_Mistook_His_Wife_for_a_Hat