RE: https://infosec.exchange/@SecureOwl/116817499565264483

It's worth reiterating, always worth reiterating:

In this situation, the LLM does not in any sense •understand• why it did what it did, and its response is not in any way an explanation or a window into how it works. Its response is not even based on whatever underlying mechanism made it start responding in a different language. Really, it's not!

Its response is a generic response to being asked “Why did you switch to a different language?,” generated from answers people typically gave to similar questions in the training dataset. That's why it answers with a stereotype about Hindi.

It's roughly the same process as a kid making up an excuse when confronted about some impulsive thing they did. First comes the action, then comes the question, then they look at the question and try to generate a reasonable-sounding response.