how to migrate mysql data to ElasticSearch realtime how to migrate mysql data to ElasticSearch realtime elasticsearch elasticsearch

how to migrate mysql data to ElasticSearch realtime


There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: https://github.com/siddontang/go-mysql-elasticsearch

Another one would be this one: https://github.com/noplay/python-mysql-replication.

Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.

UPDATE:

Here is another interesting blog article on the subject: How to keep Elasticsearch synchronized with a relational database using Logstash


The best open source solution would be this.You can run this as a command line and give the incremental logic too in the command.

GO through this session to get a complete idea.


I guess best option is to simply use Kafka connect plugin called debezium, and use the Mysql Connector for source, and Elastic Search sink connector