|Pathologically Eclectic Rubbish Lister|
Memory Efficient Sparse Matrix Handlingby neversaint (Deacon)
|on Jan 26, 2009 at 15:31 UTC||Need Help??|
neversaint has asked for the
wisdom of the Perl Monks concerning the following question:
I have a text file that stores MxN (where M=3, N=4) matrix like this:
The actual dataset I have have much larger dimension, e.g.
In principle the Sparse Matrix implementation would give us this result:
Let NNZ denote the number of nonzero entries of Matrix. The first array is A, which is of length NNZ, and holds all nonzero entries of M in left-to-right top-to-bottom order. The second array is IA, which is of length m + 1 (i.e., one entry per row, plus one). IA(i) contains the index in A of the first nonzero element of row i. Row i of the original matrix extends from A(IA(i)) to A(IA(i+1)-1). The third array, JA, contains the column index of each element of A, so it also is of length NNZ.
Is there any implementation for handling such a large matrix to give us A, IA and JA, without having have to slurp all the files into RAM?
I have tested slurping it, my 4GB ram fails. The existing implementation with Math::MatrixSparse or PDL, all memory consumptive.
neversaint and everlastingly indebted.......