Turns out that if nginx doesn't encounter a matching server block, it will send the request to the first server block. So the solution was to set up a default server block that drops every request like this:
server { listen 80 default_server; return 444;}