Read in all csv files from a directory using Python
That's how I'd do it:
import osdirectory = os.path.join("c:\\","path")for root,dirs,files in os.walk(directory): for file in files: if file.endswith(".csv"): f=open(file, 'r') # perform calculation f.close()
Using pandas and glob as the base packages
import globimport pandas as pdglued_data = pd.DataFrame()for file_name in glob.glob(directoryPath+'*.csv'): x = pd.read_csv(file_name, low_memory=False) glued_data = pd.concat([glued_data,x],axis=0)
I think you look for something like this
import globfor file_name in glob.glob(directoryPath+'*.csv'): x = np.genfromtxt(file_name,delimiter=',')[:,2] # do your calculations
Edit
If you want to get all csv
files from a folder (including subfolder) you could use subprocess
instead of glob
(note that this code only works on linux systems)
import subprocessfile_list = subprocess.check_output(['find',directoryPath,'-name','*.csv']).split('\n')[:-1]for i,file_name in enumerate(file_list): x = np.genfromtxt(file_name,delimiter=',')[:,2] # do your calculations # now you can use i as an index
It first searches the folder and sub-folders for all file_names using the find
command from the shell and applies your calculations afterwards.