Scrapy - Database choice [closed] Scrapy - Database choice [closed] elasticsearch elasticsearch

Scrapy - Database choice [closed]


That very much depends on what you are planning to do with the scraped data later on.

Elasticsearch does some complex indexing operations upon insertion which will make subsequent searches in the database quite fast ... but this also costs processing time and introduces a latency.

So to answer your question whether Elasticsearch was a good choice:

If you plan to build some kind of search engine later on, Elasticsearch was a good choice (as the name indicates). But you should have a close look at the configuration of Elasticsearch's indexing options etc to make sure it works the way you need it.

If on the other hand you just want to store the data and do processing tasks on it later, Elasticsearch was a poor choice and you would be better off with Cassandra or another NoSQL database.

Which NoSQL database suits your needs best, depends - again - on the actual usage scenario.