how to send JSON object to kafka from python client
The below is my code for producer to kafka. The only thing i did differently was to use yaml.safe_load to load the json content. It returns the contents as strings instead of unicode. The following is the snippet
with open('smaller_test_prod.txt') as f: for line in f: d = yaml.safe_load(line) jd = json.dumps(d) producer.send_messages(b'zeus_metrics',jd)In here every line is a json data stored in a file.
Kafka expects the values in bytes
b`some json message`Here is my simple Kafka producer which sends the message to Kafka server.
import jsonfrom bson import json_utilfrom kafka import KafkaProducerproducer = KafkaProducer(bootstrap_servers='localhost:9092')for i in range(10): data = { 'tag ': 'blah', 'name' : 'sam', 'index' : i, 'score': {'row1': 100, 'row2': 200 } } producer.send('orders', json.dumps(data, default=json_util.default).encode('utf-8'))Here json.dumps() convert json into string and encode('utf-8') converts string into byte array.