Hive Data to Pandas Data frame

pd.read_sql() (pandas 0.24.0) takes a DB connection. Use PyHive connection directly with pandas.read_sql() as follows:

from pyhive import hiveimport pandas as pd# open connectionconn = hive.Connection(host=host,port= 20000, ...)# query the table to a new dataframedataframe = pd.read_sql("SELECT id, name FROM test.example_table", conn)

Dataframe's columns will be named after the hive table's. One can change them during/after dataframe creation if needed:

• via HiveQL: SELECT id AS new_column_name ...
• via columns attribute in pd.read_sql()

You can try this: ( I'm pretty sure it will work)

res = cur.getSchema()description = list(col['columnName'] for col in res)  ## for getting the column names of the table headers = [x.split(".")[1] for x in description] # for splitting the list if the column name contains a perioddf= pd.DataFrame(cur.fetchall(), columns = headers)df.head(n = 20)

As I had fetched data before and was trying to fetch again, so was getting empty Data Frame.

cur.execute(query)val=cur.fetchall()columnNames = [a['columnName'] for a in  cur.getSchema()]df=pd.DataFrame(data=val,columns=columnNames)#print dfreturn df