GridGain application that is slower than a multithreaded application on one machine

Question

I have implemented my first GridGain application and am not getting the performance improvements I expected. Sadly it is slower. I would like some help in improving my implementation so it can be faster.

The gist of my application is I am doing a brute force optimization with millions of possible parameters that take a fraction of a second for each function evaluation. I have implemented this by dividing up the millions of iterations into a few groups, and each group is executed as one job.

The relevant piece of code is below. the function maxAppliedRange calls function foo for every value in the range x, and returns the maximum, and the result becomes the maximum of all the maximums found by each job.

  scalar {
    result = grid !*~
      (for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
        yield () => maxAppliedRange(x, foo), (s: Seq[(Double, Long)]) => s.max)
  }

My code can chose between a multi-threaded execution on one machine or use several GridGain nodes using the code above. When I run the gridgain version it starts out like it is going to be faster, but then a few things always happen:

• One of the nodes (on a different machine) misses a heartbeat, causing the node on my main computer to give up on that node and to start executing the job a second time.
• The node that missed a heartbeat continues doing the same job. Now I have two nodes doing the same thing.
• Eventually, all jobs are being executed on my main machine, but since some of the jobs started later, it takes way longer for everything to finish.
• Sometimes an exception gets thrown by GridGain because a node timed out and the whole task gets failed.
• I get annoyed.

I tried setting it up to have many jobs so if one failed then it wouldn't be as big of a deal, but when I do this I end up with many jobs being executed on each node. That puts a much bigger burden on each machine making it more likely for a node to miss a heartbeat, causing everything to go downhill faster. If I have one job per CPU then if one job fails, a different node has to start over from the beginning. Either way I can't win.

What I think would work best is if I could do two things:

• Increase the timeout for heartbeats
• Throttle each node so that it only does one job at a time.

If I could do this, I could divide up my task into many jobs. Each node would do one job at a time and no machine would become overburdened to cause it to miss a heartbeat. If a job failed then little work would be lost and recovery would be quick.

Can anyone tell me how to do this? What should I be doing here?

Answer 1

I figured it out.

First, there is an xml configuration file that controls the details of how the grid nodes operate. The default configuration file is in GRIDGAIN_HOME/config/default-spring.xml. I could either edit this or copy it and pass the new file to ggstart.sh when I start the grid node. The two things I needed to add are:

    <property name="networkTimeout" value="25000"/>

which sets the timeout for network messages to 25 seconds, and

   <property name="executorService">
        <bean class="org.gridgain.grid.thread.GridThreadPoolExecutor">
            <constructor-arg type="int" value="1"/>
            <constructor-arg type="int" value="1"/>
            <constructor-arg type="long">
                <util:constant static-field="java.lang.Long.MAX_VALUE"/>
            </constructor-arg>
            <constructor-arg type="java.util.concurrent.BlockingQueue">
                <bean class="java.util.concurrent.LinkedBlockingQueue"/>
            </constructor-arg>
        </bean>
    </property>

The first two constructor arguments are for 1 thread to start and a max thread size of 1. The executor service controls the threadpool that executes the gridgain jobs. The default is 100, which is why my application was being overwhelmed and the heartbeats were being timed out.

The other change I had to make to my code is:

  scalar.apply("/path/to/gridgain home/config/custom-spring.xml") {
    result = grid !*~
      (for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
        yield () => maxAppliedRange(x, kalmanBruteForceObj.performKalmanIteration), (s: Seq[(Double, Long)]) => s.max)
  }

Because without the .apply statement it starts a grid node with all default options, not the configuration file with the above edits, which is what I want.

Now it works exactly as I need it to. I can divide up the task into small pieces and even my weakest and slowest computer can make a contribution to this effort.