How can I concat multiple dataframes in Python? [duplicate]

I think you can just put it into a list, and then concat the list. In Pandas, the chunk function kind of already does this. I personally do this when using the chunk function in pandas.

pdList = [df1, df2, ...]  # List of your dataframesnew_df = pd.concat(pdList)

To create the pdList automatically assuming your dfs always start with "cluster".

pdList = []pdList.extend(value for name, value in locals().items() if name.startswith('cluster_'))

Generally it goes like:

frames = [df1, df2, df3]result = pd.concat(frames)

Note: It will reset the index automatically.Read more details on different types of merging here.

For a large number of data frames:If you have hundreds of data frames, depending one if you have in on disk or in memory you can still create a list ("frames" in the code snippet) using a for a loop. If you have it in the disk, it can be easily done just saving all the df's in one single folder then reading all the files from that folder.

If you are generating the df's in memory, maybe try saving it in .pkl first.

Use:

pd.concat(your list of column names)

And if want regular index:

pd.concat(your list of column names,ignore_index=True)