Turns out: HAproxy on the server side kept crashing for god knows how long now after I removed a domain one of the backends pointed towards, now unable to be resolved. But since it only handled IPv4 connections, those were the only ones that broke.

At the same time, IPv6 in my home network seems to have fallen over with an unrelated cause and at work, we just don't have IPv6 connectivity. But most mobile providers do offer IPv6 per default, so those connections would work.

Arggghhh!