I was doing a bit more thinking about this today and I reckon the thing that irks me the most is that the tools already exist to reliability and tractably write correct software for some pretty damn strong definitions of 'correct'. But, the industry has for years dragged its feet and has systematically decided that safety and privacy are worth throwing under the bus in order to prioritise extracting value at pace. In short, there's a lot of laziness, and it's been left to researchers and hobbyists to push the dial on a lot of this stuff.
What AI seems to offer to industry is a shortcut: skip investing in the time and skills and workers required to get stuff right, and instead crank the magic post-hoc proof machine to get a nice rubber stamp for free. That whatever is spat out isn't nearly a strong or useful enough guarantee for the application is an irrelevance: the company gets to claim that it's been diligent and the responsibility for failure gets shifted into the machine instead.