Problem of your example.

Trying your code on small scale, I notice even if you set dtype=int, you are actually ending up with dtype=object in your resulting dataframe.

header = ['a','b','c']rows = 11df = pd.DataFrame(columns=header, index=range(rows), dtype=int)df.dtypesa    objectb    objectc    objectdtype: object

This is because even though you give the pd.read_csv function the instruction that the columns are dtype=int, it cannot override the dtypes being ultimately determined by the data in the column.

This is because pandas is tightly coupled to numpy and numpy dtypes.

The problem is, there is no data in your created dataframe, thus numpy defaults the data to be np.NaN, which does not fit in an integer.

This means numpy gets confused and defaults back to the dtype being object.

Problem of the object dtype.

Having the dtype set to object means a big overhead in memory consumption and allocation time compared to if you would have the dtype set as integer or float.

Workaround for your example.

df = pd.DataFrame(columns=header, index=range(rows), dtype=float)

This works just fine, since np.NaN can live in a float. This produces

a    float64b    float64c    float64dtype: object

And should take less memory.

More on how to relate to dtypes

See this related post for details on dtype:Pandas read_csv low_memory and dtype options

The similar problem i had faced with 3 GB data today and i just did little change in my coding style like instead of file.read() and file.readline() method i used below code, that below code just load 1 line at a time in ram

import redf_list = []with open("ms.txt", 'r') as f:    for line in f:        #process(line)        line = line.strip()        columns = re.split("\t", line, maxsplit=4) # you should modify these according to your split criteria        df_list.append(columns)

Here is code to convert your data into pandas dataframe.

import pandas as pddf = pd.DataFrame(df_list)# here you will have to modify according to your data frame needs