Java Neo4J out of memory

Question

This is a bit like this: Neo4j OutOfMemory problem

But it's outdated and apparently so are the solutions as far as I can tell.

So I'm trying to insert around 100K nodes with 5.5M relations (I actually cut down my data set so it's now more like <100K nodes with 2.8M relations).

After a while, it runs out of memory and I get an exception like so:

Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.StringBuilder.toString(Unknown Source)
    at org.neo4j.kernel.impl.util.StringLogger$ActualStringLogger.logMessage(StringLogger.java:276)
    at org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:85)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.LinkedList.addBefore(Unknown Source)
    at java.util.LinkedList.add(Unknown Source)
    at org.neo4j.kernel.impl.nioneo.store.IdGeneratorImpl.freeId(IdGeneratorImpl.java:291)
    at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.freeId(CommonAbstractStore.java:382)
    at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doRollback(WriteTransaction.java:315)
    at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.rollback(XaTransaction.java:278)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:518)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:558)
    at org.neo4j.kernel.impl.transaction.TxManager.rollback(TxManager.java:610)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:129)
    at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:119)
    at sqlToGraph.SqlToGraph.main(SqlToGraph.java:81)

I've tried passing -Xmx1500m to java, which is about the limit of what I can pass because before it complains about not being able to allocate the heap space. It lasts significantly longer, but still doesn't finish.

Here is the (slightly edited) code:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    while (rs.next()) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();

Answer 1

Adding another answer here. So given the code the problem is that you never commit your transaction. Transaction data is kept in memory until committed so all your created nodes and relationships will just sit in memory awaiting a commit and that's why you eventually get OOM.

I'd propose this code change:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    for (int i = 0; rs.next(); i++) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));

        // Commit every now and then to free memory.
        if ( i > 0 && i % 10000 == 0 ) {
            tx.success();
            tx.finish();
            tx = graphDb.beginTx();
        }
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();