Taking up too much memory - python

Question

I wrote a recursive function that exhaustively generates matrices of certain characteristics. The function is as such:

def heavies(rowSums,colSums,colIndex,matH):
    if colIndex == len(colSums) - 1:
        for stuff in heavy_col_permutations(rowSums,colSums,colIndex):
            matH[:,colIndex] = stuff[0]
            yield matH.copy()
        return

    for stuff in heavy_col_permutations(rowSums,colSums,colIndex):
        matH[:,colIndex] = stuff[0]
        rowSums = stuff[1]

        for matrix in heavies(rowSums,colSums,colIndex+1,matH):
            yield matrix

and heavy_col_permutations is a function that just returns a column of a matrix with characteristics I need as well.

The problem is that as heavies is yielding a lot of matrices, it takes up too much memory. I end up calling this from another function one by one, and eventually I take up too much RAM and my process is killed (I'm running this on a server with memory caps). How can I write this to make it use less memory?

The program looks something like:

r = int(argv[1])
n = int(argv[2])
m = numpy.zeros((r,r),numpy.dtype=int32)
for row,col in heavy_listing(r,n):
    for matrix in heavies(row,col,0,m):
        # do more stuff with matrix

And I know that the function heavies is where the large amount of memory sucking is happening, I just need to lessen it.

Answer 1

Things you can try:

• Ensure that the matrix copies created by heavies() are not kept referenced in memory.
• Look at the gc module, call collect() and play around with set_threshold()
• Rewrite the function to be iterative instead of recursive