This also is exactly the sort of thing I mean when I say "Please do not use computers to intensify the stupid."
E.g. instead of asking for a cure for cancer, why not ask it how we can stop giving ourselves so much fucking cancer? (And also cure cancer too, duh, but you get what I'm saying right?)
If you ask your LLM buddy "How can I improve my health?" would it tell you about Reiki (and a couple of other things I could mention.)
If it doesn't, and people are relying on it, that makes it a kind of prison of conventionality, eh?
It's intensifying the stupid rather than getting at the core problem.
Instead of replacing all human drivers with robots why not redesign things so that we don't have to drive so much? (And yeah, duh, also robot drivers, once they're reliable and don't kill people!)
Instead of making better prosthetics and wheelchairs, why not human limb and organ regeneration? (And yeah, better prosthetics and wheelchairs too.)
Instead of an AI arms race between blocs, why not ask it how we can all get along and take care of each other in cooperation and harmony?
(This one has not the "why not both?" energy, eh?)
In sum, once you get the hang of it, every single dumb or terrifying thing has this alternate "what if good idea instead" dual to it.
It just takes a little intelligence to find it, and now the machines are intelligent enough to look for mutually beneficial solutions if only we would ask them to!
(And yeah, ethics... We need to rebuild ML systems on ethical lines and use those instead of these Ouija board/slot machine fucking things.)
But again, as I'm always ALWAYS yammering on about: we already have all the solutions we need, we just have to notice that and apply them.