How reliable is ElasticSearch as a primary datastore against factors like write loss, data availability How reliable is ElasticSearch as a primary datastore against factors like write loss, data availability elasticsearch elasticsearch

How reliable is ElasticSearch as a primary datastore against factors like write loss, data availability


Short answer: it depends on your use case, but you probably don't want to use it as a primary store.

Longer answer: You should really understand all of the possible issues that can come up around resiliency and data loss. Elastic has some great documentation of these issues which you should really understand before using it as a primary data store. In addition Aphyr's post on the topic is a good resource.

If you understand the risks you are taking and you believe that those risks are acceptable (e.g. because small data loss is not a problem for your application) then you should feel free to go ahead and try it.


It is generally a good idea to design redundant data storage solutions. For example, it could be a fast and reliable approach to first just push everything as flat data to a static storage like s3 then have ES pull and index data from there. If you need more flexibility leveraging some ORM, you could have an RDS or Redshift layer in between. This way the data can always be rebuilt in ES.

It depends on your needs and requirements how you set the balance between redundancy and flexibility/performance. If there's a lot of data involved, you could store the raw data statically and just index some parts of it by ES.

Amazon Lambda offers great features:

Many developers store objects in Amazon S3 while using Amazon DynamoDB to store and index the object metadata and enable high speed search. AWS Lambda makes it easy to keep everything in sync by running a function to automatically update the index in Amazon DynamoDB every time objects are added or updated from Amazon S3.