Design : A Java Application with high throughput

Question

I have a scenario, in which

1. A HUGE Input file with a specific format, delimited with \n has to be read, it has almost 20 Million records.
2. Each Record has to be read and processed by sending it to server in specific format.

=====================

I am thinking on how to design it.
- Read the File(nio)
- The thread that reads the file can keep those chunks into a JMS queue.
- Create n threads representing n servers (to which the data is to be sent). and then n Threads running in parallel can pick up one chunk at a time..execute that chunk by sending requests to the server.

Can you suggest if the above is fine, or you see any flaw(s) :). Also it would be great if you can suggest better way/ technologies to do this.

Thank you!

Updated : I wrote a program to read that file with 20m Records, using Apache Commons IO(file iterator) i read the file in chunks (10 lines at at time). and it read the file in 1.2 Seconds. How good is this? Should i think of going to nio? (When i did put a log to print the chunks, it took almost 26seconds! )

Answer 1

20 million records isn't actually that many so first off I would try just processing it normally, you may find performance is fine.

After that you will need to measure things.

You need to read from the disk sequentially for good speed there so that must be single threaded.

You don't want the disk read waiting for the networking or the networking waiting for the disk reads so dropping the data read into a queue is a good idea. You probably will want a chunk size larger than one line though for optimum performance. Measure the performance at different chunk sizes to see.

You may find that network sending is faster than disk reading already. If so then you are done, if not then at that point you can spin up more threads reading from the queue and test with them.

So your tuning factors are:

• chunk size
• number of threads.

Make sure you measure performance over a decent sized amount of data for various combinations to find the one that works best for your circumstances.

Answer 2

I believe you could batch the records instead of sending one at a time. You could avoid unnecessary network hops given the volume of data that need to be processed by the server.