Limit retries for secured directory with .htaccess file
There's no way to do this using apache because of the way HTTP authentication works, the number retries is determined by the browser. One thing I know that people have done is write a log file parser to parse the apache logs to track the 401 responses by IP and request (remember that the first 401 is the request that prompts the login dialog) and do something when there's an excessive number of 401's from one IP for one request. Among of the things the parser can do is block the offending IP using iptables (or something similar) for a certain period of time, or modify a map file that can be used in conjunction with RewriteMap to redirect the client to a "you've failed too many times" page. The only other thing I can think of is writing a custom auth module, but writing a log parser seems much easier.