Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Re: Re: Heap sorting in perl

by dws (Chancellor)
on Apr 05, 2003 at 20:58 UTC ( #248337=note: print w/replies, xml ) Need Help??

in reply to Re: Heap sorting in perl
in thread Heap sorting in perl

I have a mod_perl process that slurps up a large dataset from a database, sorts it, then displays the first N items. This happens to balloon the memory consumption of the apache process to an unacceptable level.

An obvious question, perhaps, but have you ruled out getting the database to do the work for you? That'd be a no-brainer if the column you're interested in is indexed. I'm guessing it isn't. Still, having the database do the sort might still be faster than scanning the entire table, pulling the data across a process boundary a row at a time, and then filtering it in the Apache process.

Assuming you have a fairly large table, here's an ugly heuristic you might try: Probe the first 2*M rows of the table, sort, and find the M'th largest value. Then run a query that does a sort on the database side, subsetting the data before sorting by adding a WHERE clause to reject any row whose target column has a value greater than the M'th value you found the first time through. You can tune this by trying 3*M rows, 4*M rows, etc. The idea is to reduce the set that the database needs to sort.

Replies are listed 'Best First'.
Re3: Heap sorting in perl
by blakem (Monsignor) on Apr 06, 2003 at 08:37 UTC
    Letting the datbase do the sort is a good approach, but I was hoping for a more general, memory efficient solution that allows for arbitrary perl comparison routines.

    Its a common enough situation that I would like to create a subroutine that returns the rows I want and takes a database handle, a sorting coderef, M, and an optional offset. I'm envisioning something like:

    my $sortedrows = db_heap_sort($dbh, $coderef, $M, $offset);
    The question is, is there a perl heap implementation that would be suitable for implementing the above sub. seems like the obvious choice, but I have doubts about its current status.

    I have emailed the author, and will follow up with any new information I find out.


Re: Re: Re: Heap sorting in perl
by pg (Canon) on Apr 06, 2003 at 03:17 UTC
    If those are rows in a database, I agree with dws, just let database do it.

    If that particular column is not indexed, I suggest index it, as you need it.

    "order by" is usually not too expansive, especially when it only involves indexed range scan.

    Depends on your database, some database directly support syntax that allows to retrieve first N rows.
      If that particular column is not indexed, I suggest index it, as you need it.

      If this is a write-mostly, query-infrequently table, indexing could significantly degrade performance. For log/audit files that record large numbers of miscellaneous value, indexing the values is rarely an effective strategy.

        When this is generally correct, I don't really see a reason why a person want to "sort by" some column containing some miscellaneous values. If unfortunately that's the case, then should look into the data, and have some sort of key information extracted.

        Any way, I believe blakem has his stuff well organized, and what he is doing is reasonable.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://248337]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2018-06-21 05:37 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (117 votes). Check out past polls.