StringBuilder insufficient memory C#

Question

Here is what I am trying to achieve. Looping through a dataset over 1 million records and create a data dump in text file export to C drive.

I am looping through a dataset with over a million of records. Here is what is inside the loop

I am using a StringBuilder inside the loop.

myString.Append(ds.tables[0](i)(0));  <-- each datarow is not more than 10 char long.

It throws an error saying insufficient memory. I have 12 gb of Ram.

How do I go about fixing this problem?

Answer 1

Don't use an intermediate StringBuilder -- its contents sit in your computer's RAM before you presumably call .ToString() on it to write the result to disk. Instead, write the data to disk as you are processing it, something like:

using (var sw = new StreamWriter(outputFilePath, true)) 
{
    // start loop
        sw.Write(ds.tables[0](i)(0));
    // end loop
}

This will write text to a file using the default encoding (UTF-8) and buffer size (I think it's 4KB).

Answer 2

Why do you store the large string in memory at all? If all you want to do is to write it to a text-file you could use a StreamWriter to write in batches:

using(var writer = new StreamWriter("c:\\file.txt", true))
{
    for(int rowNum = 0; rowNum < ds.tables[0].Rows.Count; rowNum++)
    {
        DataRow row = ds.Tables[0].Rows[rowNum];
        writer.Write(row.Field<string>(0));
    }
}

But maybe you can optimze it further. Do you really need the large DataSet at all? If the data came from a database you could use a DataReader to stream it lazily. Then you can write to the text-file without memory.