Instruct Scrapy to ignore the content length of a site
You can set the maximum response size that Scrapy will download to 'unlimited' using download_maxsize=0 at the command line (using scrapy crawl myspider -s download_maxsize=0). The default is 1024 bytes.
You can also do this per-spider in their scrapy.spiders.Spider.custom_settings attribute, per-project in the myproject.settings module of a Scrapy project, or globally within the scrapy.settings.default_settings module.
See the official Scrapy documentation for this option here or further instructions on built-in settings here.