Python: Copy identical .csv files from various folders (each folder has one .csv file) into a single folder

Question

The file system is designed as follows in my problem:

Main folder -> sub-folder -> data.csv

Here, the Main folder has several sub-folders where each sub-folder has a single data.csv file (note that .csv files in every sub-folder has same file name as data.csv) . For data processing, I need all the .csv files to be in a single folder so that I can use the following code (the following code works when I manually copy and rename (since all .csv files have identical name) each .csv file from sub-folders into a single folder). While renaming the .csv files, I used the sub-folder's name, for example: if a sub- folder is named as a1, I rename the .csv file inside this sub-folder as a1.csv.

import pandas as pd
import csv
import glob
import os

path = r'C:\Santobedi' # file path to the folder that contains manually copied and uniquely named .csv files
all_files = glob.glob(path + "/*.csv")
print(all_files)

for file in all_files:
    file_name = os.path.split(file)[-1]
    file_name_path = os.path.join('C:\Santobedi\CNN' + file_name)
    transpose_df = pd.read_csv(file).T
    transpose_df.to_csv(file_name_path)

Manually copying and renaming huge files is tiresome. I want to do it in a single go. How can I do it with Python (pandas)?

Answer 1

import os
import shutil
MAIN_FOLDER = 'main_folder'
for item in os.listdir(MAIN_FOLDER):
    if os.path.isdir(os.path.join(MAIN_FOLDER, item)):
        source = os.path.join(MAIN_FOLDER, item, 'data.csv')
        destination = os.path.join(MAIN_FOLDER, f'{item}.csv')
        shutil.move(source, destination)