aiohttp concurrent GET requests lead to ClientConnectorError(8, 'nodename nor servname provided, or not known')

After some further investigation, this issue does not appear to be directly caused by aiohttp/asyncio but rather limitations/limits stemming from both:

• The capacity/rate-limiting of your DNS Servers
• The max number of open files at the system level.

Firstly, for those looking to get some beefed-up DNS servers (I will probably not go that route), the big-name options seem to be:

• 1.1.1.1 (Cloudflare)
• 8.8.8.8 (Google Public DNS)
• Amazon Route 53

(Good intro to DNS for those like me for whom network concepts are lacking.)

The first thing that I did was to run the above on a beefed-up AWS EC2 instance - h1.16xlarge running Ubuntu which is IO optimized. I can't say this in itself helped, but it certainly cannot hurt. I'm not too familiar with the default DNS server used by an EC2 instance, but the OSError with errno == 8 from above went away when replicating the above script.

However, that presented a new exception in its place, OSError with code 24, "Too many open files." My hotfix solution (not arguing this is the most sustainable or safest) was to increase the max file limits. I did this via:

sudo vim /etc/security/limits.conf# Add these linesroot    soft    nofile  100000root    hard    nofile  100000ubuntu    soft    nofile  100000ubuntu    hard    nofile  100000sudo vim /etc/sysctl.conf# Add this linefs.file-max = 2097152sudo sysctl -psudo vim /etc/pam.d/commmon_session# Add this linesession required pam_limits.sosudo reboot

I am admittedly feeling around in the dark, but coupling this with asyncio.Semaphore(1024) (example here) led to exactly 0 of the either 2 exceptions above being raised:

# Then call this from bulk_get with asyncio.Sempahore(n)async def bounded_get(sem, url, session) -> str:    async with sem:        return await get(url, session)

Of the ~25k input URLs, only ~100 GET requests returned exceptions, mainly due to those websites being legitimately broken, with total time to completion coming in within a few minutes, acceptable in my opinion.