Re: Heap sorting in perl

[blakem]

The responses so far are very helpful, but I probably should have given a bit more context.....

The resource I am optimizing for here is memory. I have a mod_perl process that slurps up a large dataset from a database, sorts it, then displays the first N items. This happens to balloon the memory consumption of the apache process to an unacceptable level.

One approach to this, of course, is to sort the data in the database, but in this case I need the ability to define the sort function as arbritrary perl code.

Heap sorting is attractive because selecting the smallest M elements from a dataset of size N (N being much larger than M) requires storing only M elements in memory at any one time. This would be a huge win for me.

A heap is a datastructure that makes two operations very efficient: inserting a new element, and selecting (or removing) the largest element. The pseudo-code for my algorithm would look something like this:

# select the smallest $M elements;
while ($new_element = $dbh->fetchrow) {
  if ($heap->size < $M) {               # build a heap of size M
    $heap->insert($new_element);
  } elsif ($element < $heap->largest) { # maintain a heap containing
                                        # the smallest M elements 
                                        # seen so far
    $heap->remove_largest;
    $heap->insert($element);
  }
}

At the end of this loop we will have a $heap containing the M smallest elements, and never have to store more than M elements in memory at once.

Therefore, I'm looking for a decent heap implementation and my first impression of Heap.pm was somewhat discouraging. I have several paths I can take from here, but before I go hacking up Heap.pm I was wondering if there might be a better place to start.

-Blake