So it turns out that if you're using ffmpeg without actively including hardware acceleration, you're setting yourself up for a bad time. Obvious, I'm sure. But I didn't realise until a few minutes ago that ffmpeg doesn't use the GPU automatically, and that you have to ask it to.
Just run the conversion on the same video file as before on my work PC with its integrated Intel i3 GPU which converted at 6.4x speed.
I'm actually blown away.