Extracting data using U-SQL file set pattern when silent switch is true

Question

I want to extract data from multiple file so I am using file set pattern that requires one virtual column. Because of some issues in my data, I also require silent switch other wise I am not able to process my data. It looks like, when I use virtual column with silent switch it does not extract any row.

@drivers =
EXTRACT name string,
        age string,
        origin string
FROM "/input/{origin:*}file.csv"
USING Extractors.Csv(silent:true);

Note that I can extract data from a single file by removing virtual column. Is there any solution for this problem?

Answer 1

first you do not need to name the wildcard (and expose a virtual column) if you do not plan on referring to the value. Although we recommend that you make sure that you are not processing too many files with this pattern, so best may be to use the virtual column as a filter to restrict the number of files to a few thousand right now until we improve the implementation to work on more files.

I assume that at least one file contains some rows with two columns? If that is the case I think you found a bug. Could you please send me a simple repro (one file that works, and an additional file where it stops working and the script) to my email address so I can file it and we can investigate it?

Thanks!