Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Reading a directory of csv's into a Pandas Dataframe

I have a directory with csvs whose filenames represent the id of a row of a database.

I would like to read in this directory into a Pandas dataframe and join it to an existing dataframe.

Is there any way in Python to read the results of an 'ls' command into a pandas Dataframe?

I've tried getting a string of the filenames with the code below but I'm having trouble figuring out how to get it into a dataframe after.

import os

files = ''
for root, dirs, files in os.walk("."):
    for filename in files:
        files += filename
like image 220
fness Avatar asked Dec 14 '25 04:12

fness


1 Answers

You are able to walk the files, now you just need to read the csv and concat them onto a dataframe.

import os

import pandas as pd


df = None

for root, dirs, files in os.walk('.'):
    for filename in files:
        if not df:
            df = pd.read_csv(filename)
            df['filename'] = filename
            continue
        tmp = pd.read_csv(filename)
        tmp['filename'] = filename
        df = pd.concat(df, tmp)
like image 136
Rob Avatar answered Dec 15 '25 19:12

Rob