Scrapy - Output to Multiple JSON files

python json scrapy

I don't think built-in feed exporters support writing into multiple files.

One option would be to export into a single file in jsonlines format basically, one JSON object per line which is convenient to pipe and split.

Then, separately, after the crawling is done, you can read the file in the desired chunks and write into separate JSON files.

So I could then upload them to Amazon Cloud Search for indexing.

Note that there is a direct Amazon S3 exporter (not sure it helps, just FYI).