I imagine you could have a setup where for each device that needs a Linux driver you launch a very small VM with an extremely barebones kernel and a very small initramfs containing the necessary module(s) and firmware and a small /init that just proxies syscalls to the host.

It would probably end up using a lot of memory anyway but maybe samepage merging could reduce that problem.