Why is my server slow and SSL mismatching when combining IPv4 with IPv6

The question has so much stuff, it's not even clear what's being asked — all those non-server-log 404 errors don't mean much to most readers, and, certainly, I've never heard of the mere presence of IPv6 having any sort of effects on image requests suddenly returning 404 errors.

You've already provided find /etc/nginx/{conf.d,sites-enabled} and find /etc/nginx/{conf.d,sites-enabled} | xargs fgrep listen, and there's nothing out-of-the-ordinary there — it seems that every non-IPv4 listen is accompanied by an IPv6 listen. http://nginx.org/r/listen.

The next step, IMHO, would be to reproduce the error directly with curl alone, including using some combination of the -v, -4 and -6 flags to see what's going on. Make absolute sure you're doing it from the same client host where you're actually experiencing the errors you're trying to correct; this whole issue might as well stem from something being amiss on the client side.

Another useful feature of curl would be to use the domain-mapping if you think your issues are related to DNS. You can already see which IP address the request would go to if you use the -v option with curl (which may reveal the stale DNS easily, unless it's only stale within your browser); the next step is to possibly override the IP address with the --resolve option, e.g., --resolve test.vuyk.eu:443:2a03:b0c0:0:1010::190:6001 or some such.

Finally, if you're still experiencing delay and failure only over IPv6 (-6) and not via IPv4 (-4), then the issue might be related to the IPv6 connection on the client; specifically, if your connection goes through a tunnel or 6rd, then perhaps MTU discovery is broken along the path; on Linux, you could try using tracepath to troubleshoot, it prints pmtu values along the path.

FWIIW, I've tried accessing your site via an IPv6 native connection via TMo US, with MTU of 1500 for IPv6 (they actually tunnel IPv4 through IPv6, so, IPv4 MTU is 1472 on T-Mo US, from what I understand), and I couldn't see any obvious issues (and, as far as I see, you do currently have IPv6 records published for test.vuyk.eu). Perhaps the whole issue is due to stale DNS cache on the client side in your setup? What are you seeing in server logs for all those 404 errors that you see in the client?!