That's interesting - the usual "breaking RSA-4096 takes centuries", but what if you optimise by waiting for better hardware? Supposing big, universal quantum computers never make it...
Taking Moore into account, wait for 120 years, and then calculate for 3 years on the latest hardware.
But: you also have to take into account the energy use! And this will hit even worse: for RSA-4096, taking the "Landauer floor" of the energy required for a bit-flip, you need 5 years of ALL sunlight of earth!