Read CSV data in batches and Process it

Question

I have a csv file that looks like this

#DELTA,1#    
Risk1,10
Risk2,10
Risk3,10
Risk4,10
Risk5,10
#DELTA,1#    
Risk6,10
Risk7,10
Risk8,10
Risk9,10
Risk10,10

and so on. These are very large files (in order of GBs).

What I want to be able to do is to read them in batches like

start streamreader from csv file from first line to just before next #Delta starts

---Batch 1---
#DELTA,1
Risk1,10
Risk2,10
Risk3,10
Risk4,10
Risk5,10
--Batch 2-----
#DELTA,1
Risk6,10
Risk7,10
Risk8,10
Risk9,10
Risk10,10
----------------------

and once I get a batch put this subset for processing and come back and restart preparing another batch and so on till the end of file is reached.

I have tried making the LINQ's take and take while but with my understanding of LINQ I am not getting far.

Basically in summary it have to stream data in batches based on a pattern in my stream.. maybe my brain cells are dead or maybe it is too late in evening. Really appreaicate anyone's help

Answer 1

The easiest approach would be a TextReader and ReadLine().

For positioning, I would just leave the Reader open between processing the batches. If that's not an option, save the (stream) Position and restore it later.

With a StreamReader, if you have to close the file, you'd have to keep a lineCount and read-and-skip from the beginning again. Not too attractive.