StreamWriter - Process cannot access the file because it is being used by another process

Question

I am trying to rewrite an Add-Content script as a StreamWriter version, reason being that the file is ~140 MB and Add-Content is far too slow.

This is my Add-Content version, which loops through each row until it can find a header row starting FILE| and creates a new file with a filename of the second delimited (by pipe) value in that row. The Add-Content works as intended, but is really slow. It takes 35-40 mins to do it:

Param(
    [string]$filepath = "\\fileserver01\Transfer",
    [string]$filename = "sourcedata.txt"
)

$Path = $filepath
$InputFile = (Join-Path $Path $filename)
$Reader = New-Object System.IO.StreamReader($InputFile)

while (($Line = $Reader.ReadLine()) -ne $null) {
    if ($Line -match 'FILE\|([^\|]+)') {
        $OutputFile = "$($matches[1]).txt"
    }
    Add-Content (Join-Path $Path $OutputFile) $Line
}

I've researched that StreamWriter should be faster. Here is my attempt, but I get the error

The process cannot access the file '\fileserver01\Transfer\datafile1.txt' because it is being used by another process.

Param(
    [string]$filepath = "\\fileserver01\Transfer",
    [string]$filename = "sourcedata.txt"
)

$Path = $filepath
$InputFile = (Join-Path $Path $filename)
$Reader = New-Object System.IO.StreamReader($InputFile)

while (($Line = $Reader.ReadLine()) -ne $null) {
    if ($Line -match 'FILE\|([^\|]+)') {
        $OutputFile = "$($matches[1])"
    }
    $sw = New-Object System.IO.StreamWriter (Join-Path $Path $OutputFile)
    $sw.WriteLine($line)
}

I assume it's something to do with using it in my loop.

Sample data:

FILE|datafile1|25/04/17
25044|0001|37339|10380|TT75
25045|0001|37339|10398|TT75
25046|0001|78711|15940|TT75
FILE|datafile2|25/04/17
25047|0001|98745|11263|TT75
25048|0001|96960|13011|TT84
FILE|datafile3|25/04/17
25074|0001|57585|13639|TT84
25075|0001|59036|10495|TT84
FILE|datafile4|25/04/17
25076|0001|75844|13956|TT84
25077|0001|17430|01111|TT84

Desired outcome is 1 file per FILE| heade row using the second delimited value as the file name.

Answer 1

You're creating the writer inside the while loop without ever closing it, thus your code is trying to re-open the already opened output file with every iteration. Close an existing writer and open a new one whenever your filename changes:

while (($Line = $Reader.ReadLine()) -ne $null) {
    if ($Line -match 'FILE\|([^\|]+)') {
        if ($sw) { $sw.Close(); $sw.Dispose() }
        $sw = New-Object IO.StreamWriter (Join-Path $Path $matches[1])
    }
    $sw.WriteLine($line)
}
if ($sw) { $sw.Close(); $sw.Dispose() }

Note that this assumes that you won't open the same file twice. If the same output file can appear multiple times in the input file you need to open the file for appending. In that case replace

$sw = New-Object IO.StreamWriter (Join-Path $Path $matches[1])

with

$sw = [IO.File]::AppendText((Join-Path $Path $matches[1]))

Note also that the code doesn't do any error handling (e.g. input file doesn't begin with a FILE|... line, input file is empty, etc.). You may want to change that.