Scrapy Python Set up User Agent
Move your USER_AGENT line to the settings.py file, and not in your scrapy.cfg file. settings.py should be at same level as items.py if you use scrapy startproject command, in your case it should be something like myproject/settings.py
Just in case anyone lands here that manually controls the scrapy crawl. i.e. you do not use the scrapy crawl process from the shell...
$ scrapy crawl myprojectBut insted you use CrawlerProcess() or CrawlerRunner()...
process = CrawlerProcess()or
process = CrawlerRunner()then the user agent, along with other settings, can be passed to the crawler in a dictionary of configuration variables.
Like this...
process = CrawlerProcess( { 'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' } )
I had the same problem. Try running your spider as superuser. I was running the spider directly with the command "scrapy runspider", when I just tried executing it with "sudo scrapy runspider" it worked.