IPTables do not block IP with ipset immediately

The reason that a firewall rule may have no immediate effect on blocking traffic may be due to stateful inspection of packets.

It may be inefficient for the firewall to analyse every single packet that arrives in the line, so, for performance reasons, what happens is that the rules the user creates often apply only to the initial packets that establish the connection (known as TCP's SYN, SYN+ACK, ACK) — subsequently, said connection is automatically whitelisted (to be more precise, it is the state that the original rule has created that is whitelisted), until terminated (FIN).

What likely happens here is that, due to pipelining and keep-alive connections, which nginx excels at, a single connection may be used to issue and process multiple independent HTTP requests.

So, in order for you to fix the issue, you could either disable pipelining and keep-alives in nginx (not a good idea, as it'll affect performance), or drop the existing whitelisted connections, e.g., with something like tcpdrop(8) on *BSD — surely there must be a Linux equivalent tool, too.

However, if you're simply having an issue with a single client performing too many requests, and as such overloading your backend, then the appropriate course of action may be to rate-limit the clients based on the IP-address, with the help of the standard limit-req directive of nginx. (Note, however, that some of your customers may be behind a carrier-grade NAT, so, be careful with how much you apply the limiting to ensure false-positives won't be an issue.)

Have you looked at this post here: https://serverfault.com/questions/523021/why-is-iptables-not-blocking-an-ip-address

This post shows shows the person's ip getting dropped using -A INPUT -p tcp --dport 80 -j LOG --log-prefix "HTTP: "

I haven't run into this issue personally so please let me know if this is helpful, good luck.