Flask - Pulling the live stream kafka data - Integrating Kafka with Python Flask

I would check out the Kafka package for python:

http://kafka-python.readthedocs.org/en/master/usage.html

This should get you setup to stream data from Kafka. Additionally, I might check out this project: https://github.com/travel-intelligence/flasfka which has to do with using Flask and Kafka together (just found it on a google search).

I'm working on a similar problem (small Flask app with live streaming data coming out of Kafka).

You have to do a couple things to set this up. First, you need a KafkaConsumer to grab messages:

from kafka import KafkaConsumerconsumer = KafkaConsumer(group_id='groupid', boostrap_servers=kafkakserver)consumer.subscribe(topics=['topicid'])try:    # this method should auto-commit offsets as you consume them.    # If it doesn't, turn on logging.DEBUG to see why it gets turned off.    # Not assigning a group_id can be one cause    for msg in consumer:        # TODO: process the kafka messages.finally:    # Always close your producers/consumers when you're done    consumer.close()

This is about the most basic KafkaConsumer. The for loop blocks the thread and loops until it's committed the last message. There is also the consumer.poll() method to just grab what messages you can in a given time, depending on how you want to architect the data flow. Kafka was designed with long-running consumer processes in mind, but if you're committing messages properly you can open and close consumers on an as needed basis as well.

Now you have the data, so you can stream it to the browser with Flask. I'm not familiar with ChartJS, but live streaming from Flask centers on calling a python function that ends in yield inside a loop instead of just a return at the end of processing.

Check out Michael Grinberg's blog and his followup on streaming as practical examples of streaming with Flask. (Note: anyone actually streaming video in a serious Web app will probably want to encode it into a video codec like the widely used H.264 using ffmpy and wrap it in MPEG-DASH ...or maybe choose a framework that does more of this stuff for you.)