Writing JSON column to Postgres using Pandas .to_sql Writing JSON column to Postgres using Pandas .to_sql postgresql postgresql

Writing JSON column to Postgres using Pandas .to_sql


I've been searching the web for a solution but couldn't find any so here is what we came up with (there might be better ways but at least this is a start if someone else runs into this).

Specify the dtype parameter in to_sql.

We went from:df.to_sql(table_name, analytics_db) to df.to_sql(table_name, analytics_db, dtype={'name_of_json_column_in_source_table': sqlalchemy.types.JSON}) and it just works.


If you (re-)create the JSON column using json.dumps(), you're all set.This way the data can be written using pandas' .to_sql() method, but also the much faster COPY method of PostgreSQL (via copy_expert() of psycopg2 or sqlalchemy's raw_connection()) can be employed.

For the sake of simplicity, let's assume that we have a column of dictionaries that should be written into a JSON(B) column:

import jsonimport pandas as pddf = pd.DataFrame([['row1',{'a':1, 'b':2}],                   ['row2',{'a':3,'b':4,'c':'some text'}]],                  columns=['r','kv'])# conversion function:def dict2json(dictionary):    return json.dumps(dictionary, ensure_ascii=False)# overwrite the dict column with json-stringsdf['kv'] = df.kv.map(dict2json)


I am unable to comment peralmq's answer, but in case of postgresql JSONB you can use

from sqlalchemy import dialectsdataframe.to_sql(..., dtype={"json_column":dialects.postgresql.JSONB})