@nelson @benjamineskola for a theoretical super-intelligence that is extremely good at modeling the real world, the ability to say something that sounds convincing but is false would be extremely natural, as it would be able to accurately predict when it should lie to achieve its end goals.
however, LLMs are not at all doing this. it just seems they can't model the real world, instead their model of the real world is inaccurate so they 'hallucinate' things, aka 'lie'