Airbnb Airflow vs Apache Nifi [closed] Airbnb Airflow vs Apache Nifi [closed] python python

Airbnb Airflow vs Apache Nifi [closed]


For a great overview of Airflow and Apache NiFi checkout this reddit post: https://www.reddit.com/r/bigdata/comments/51mgk6/comparing_airbnb_airflow_and_apache_nifi/

For your specific use-case of ingesting Json files, enriching them and routing them to Kafka I believe NiFi is the right tool for the job. A couple of processors you could potentially use, as well as documentation for each, are below:

GetFile: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.GetFile/index.html

JoltTransformJSON: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.JoltTransformJSON/index.html

PublishKafka (or PublishKafka_0_10 depending on your version): https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.9.2/org.apache.nifi.processors.kafka.pubsub.PublishKafka/index.html