Elastic-Search scroll(scan) into Pandas DataFrame
You can use panda's json_normalize.
from pandas.io.json import json_normalizefrom elasticsearch import Elasticsearchfrom elasticsearch.helpers import scan as escanimport pandas as pdes = Elasticsearch(dpl_server, verify_certs=False)body = { "size": 1000, "query": { "match_all": {} }}response = escan(client=es, index="index", query=body, request_timeout=30, size=1000)# Initialize a double ended queueoutput_all = deque()# Extend deque with iteratoroutput_all.extend(response)# Convert deque to DataFrameoutput_df = json_normalize(output_all)Here you can find more info on the double ended queue.