Get all files names in subfolders Azure Data factory

Question

I have a below Folder Structure in Data lake, I want to get all .csv file names from all subfolders of my ParentFolder directory. All my files are .csv files is there a simple approach to do using Metadata activity.

ParentFolder > Year=2020Folder

          2020-10-20Folder > 2020-10-20.csv

          2020-10-21Folder > 2020-10-21.csv

          2020-10-22Folder > 2020-10-22.csv

Answer 1

I've made a test to get the FileNames successfully.

I created the same file structure as yours.

1. In ADF, we can define an Array type variable to store the file names later.

2. It's the summary of the pipeline.

At the GetMetaData1 activity, let's define a DataSet of the root folder 2020Folder. Then we use Child Items to get all the subfolders.

3. At the ForEach1 activity, we can use the expression @activity('Get Metadata1').output.childItems to foreach the Folder list.

4. In the ForEach1 activity, We can add dynamic content @item().name to pass the subfolder name to the GetMetadata2 activity. Then we can use the GetMetadata2 activity to get the Child Items from the subfolder.

5. At the Append variable activity, we can use the array variable FileNames we defined previously to store all the filenames. Here we use expression @activity('Get Metadata2').output.childItems[0] to get the filename.

6. In the end. We can define another Array type variable to store and review the result.

The output we can see the array.