I am using Scrapy to crawl data, but the server block my IP
it is the router's public IP which is blocked.
In this scenario there are two networks.
One, the public internet - Where the server (hosting the website you crawl) is connected.
Two, your private home network - Where your Mac is connected.
Your router acts as a gateway from your private home network to the internet and thus helps your Mac talk to the Server.
To act as a "Gateway" the router will have two IP addresses. One private IP address for your home network and one public IP address. The public IP address is what is visible to the server. In the Server's viewpoint the public IP address is from where the crawl requests are made.
Hence it is the router's public IP which is blocked.
Also please respect a website's Terms of Service and crawl responsibly.
If you don't want to get banned try the following settings in the settings.py:
- Limit CONCURRENT_REQUESTS
- Set DOWNLOAD_DELAY
Reference : http://doc.scrapy.org/en/latest/topics/settings.html
It will get the address of provided by your Internet provider, so your public IP. could check this post: Get proxy ip address scrapy using to crawl