Powershell Memory Usage

Question

Im abit of a noob to Powershell so please dont chastise me :-) So I've got some rather large log files (600mb) that I need to process, my script essentially strips out those lines that contain "Message Received" then tokenises those lines and outputs a few of the tokens to an output file.

The logic of the script is fine (although Im sure it could be more efficient) but the problem is that as I write lines to the output file and the file subseuqenly grows larger, the amount of memory that powershell utilises also increases to the point of memory exhaustion.

Can anyone suggest how I can stop this occuring? I thought about breaking up the log into a temporary file of only say 10mb then processing on the temp file instead?

Heres my code, any help you guys could give would be fantastic :-)

Get-Date | Add-Content -Path d:\scripting\logparser\testoutput.txt


$a = Get-Content D:\scripting\logparser\importsample.txt 


foreach($l in $a){
#$l | Select-String -Pattern "Message Received." | Add-Content -Path d:\scripting\logparser\testoutput.txt
if
    (($l | Select-String -Pattern "Message Received." -Quiet) -eq "True")

    {
    #Add-Content -Path d:\scripting\logparser\testoutput.txt -value $l
    $var1,$var2,$var3,$var4,$var5,$var6,$var7,$var8,$var9,$var10,$var11,$var12,$var13,$var14,$var15,$var16,$var17,$var18,$var19,$var20 = [regex]::split($l,'\s+')
    Add-Content -Path d:\scripting\logparser\testoutput.txt -value $var1" "$var2" "$var3" "$var4" "$var16" "$var18

    }
else
    {}
}   
Get-Date | Add-Content -Path d:\scripting\logparser\testoutput.txt

Answer 1

If you do everything in the pipe, only one object at a time (one line from the file in your case) needs to be in memory.

Get-Content $inputFile | Where-Object { $_ -match "Message Received" } |
  foreach-object -process {
  $fields = [regex]::split($_,'\s+') # An array is created
  Add-Content -path $outputFile -value [String]::Join(" ", $fields[0,1,2,3,15,17])
}

The $fields[0,1,2,3,15,17] creates an array of the given indices of $fields.

This could also be done in a single pipeline using an expression rather than a property name passed to Select-Object, but would be less clear.

Answer 2

a working powershell example:

$csvFile = "c:\test.txt"

$file_reader = [System.IO.File]::OpenText($csvFile)
$row = "";
while(($row = $file_reader.ReadLine()) -ne $null) 
{ 
    # do something with '$row' 
    Write-Host row: $row
}
$file_reader.Close()