Just to share, well, I just found out that the reason I spent about three hours trying to get dropbear-initramfs to accept a key when using ssh is because you must convert the key running /usr/lib/dropbear/dropbearconvert from the dropbear format to the openssh format or vice versa depending on which you used to generate it or you get a libcrypto error and not one single guide I followed anywhere mentioned this...
As a side note, a lot of ssh users probably already know this, but I finally figured out the most reasonable way to deal with the fact that the key is different between the dropbear server and the normal ssh server is to just use a manually set known_hosts file like so:
ssh -o UserKnownHostsFile=~/.ssh/known_hosts.initramfs -i ~/.ssh/dropbear-server.key root@[address]