Computing generalized eigen values for sparse matrices in python

Question

I am using scipy.sparse.linalg.eigsh to solve the generalized eigen value problem for a very sparse matrix and running into memory problems. The matrix is a square matrix with 1 million rows/columns, but each row has only about 25 non-zero entries. Is there a way to solve the problem without reading the entire matrix into memory, i.e. working with only blocks of the matrix in memory at a time?

It's ok if the solution involves using a different library in python or in java.

Answer 1

For ARPACK, you only need to code up a routine that computes certain matrix-vector products. This can be implemented in any way you like, for instance reading the matrix from the disk.

from scipy.sparse.linalg import LinearOperator

def my_matvec(x):
    y = compute matrix-vector product A x
    return y

A = LinearOperator(matvec=my_matvec, shape=(1000000, 1000000))
scipy.sparse.linalg.eigsh(A)

Check the scipy.sparse.linalg.eigsh documentation for what is needed in the generalized eigenvalue problem case.

The Scipy ARPACK interface exposes more or less the complete ARPACK interface, so I doubt you will gain much by switching to FORTRAN or some other way to access Arpack.