Re^3: An efficient, scalable matrix transformation algorithm

[dHarry]

I am still not convinced you have to transpose the matrix but maybe I am having an off-day, i.e. to blind to see it;-) When I read to your post the first thing that jumps into mind is the use of a database., especially since your "reduce functions" are simple.

However, assuming you insist on transposing, you might be able to speed things up. The case N != M is more difficult then N = M. The approach you take right now is the straightforward approach but this can give poor performance (as you have probably noticed).

In general: depending on circumstances, things like how much storage you have available and more importantly how M relates to N (|N-M| small or gcd(N,M) is not small) you can improve by using a more refined algorithm. (This I have taken from the Wiki, already suggested in my earlier response.) You can find pointers there to articles on the subject of in-place matrix transposition. I don’t know of any Perl implementations but the pointers on the Wiki also contain source code so you should be able to rewrite that into Perl if needed.

HTH, dHarry

[Luftkissenboot]

Well, this source is the degenerate case where the objects holding the records are turned into simple arrayrefs, to illustrate the algorithm. I'm willing and able to refactor, though, of course.

The transposition happens because the reductions need to be applied to columns, and the functions sum(), avg(), etc. operate on lists. The only alternative to the full transposition I can see would be a column splice -> list/array, which essentially results in the same thing.

i.e. I don't see a way to avoid the transposition, one way or another ... ? Maybe it's me that's missing something?

[dHarry]

It was the following text that triggered me (again from the Wiki):

On a computer, one can often avoid explicitly transposing a matrix in memory by simply accessing the same data in a different order...

Because your reduction functions are simple (not like some Fourier transformation) I thought you might get away with this. You could also consider to change your data structure.