a more efficient lexicographical sort?

[Anonymous Monk]

Is anyone aware of a more efficient method to sort 500,000 plus OIDs other then using the Net::SNMP method: oid_lex_sort. I’ve tried it using it with both hash keys and arrays but it still takes around 3 – 5 mins and 400 megs of memory.

Thanks

[demerphq]

I have a feeling using perl's sort might not be the greatest plan in this situation. I'm sure there is a module that does an inplace sort that will be faster (I've not needed it so i haven't looked). Pushing 500k items on the stack to sort them and then popping them back off is IMO going to really hurt for long lists. An inline sort would avoid that overhead. The actually oid_lex_sort() code actually does this an extra time. In my implementation I avoid part of it by passing the array in as a ref.

With regard to the actual implementation of ST that oid_grt_sort uses, I think the GRT might win. Its hard to say. Avoiding the array creation I should think will help with speed and I know will help for memory footprint. Overall I'd guess that this would be faster, but I haven't benchmarked it at all. Actually for that matter I've minimally tested this, but it appears to work and should give you an idea how it could be done. Also I've included the code for oid_lex_sort() which really you should have done yourself. :-)

use strict;
use warnings;

sub oid_lex_sort(@)
{
   return @_ unless (@_ > 1);

   map  { $_->[0] }
   sort { $a->[1] cmp $b->[1] }
   map  {
      my $oid = $_;
      $oid =~ s/^\.//o;
      $oid =~ s/ /\.0/og;
      [$_, pack('N*', split('\.', $oid))]
   } @_;
}

sub oid_grt_sort
{
   my $ary=shift;
   map  {
          my $len=unpack('N*',substr($_,-4));
          substr($_,-$len-4,$len)
        }
   sort
   map  {
      my $oid=$_;
      $oid =~ s/^\.//;
      $oid =~ s/ /.0/g;

      pack('N*', split(/\./, $oid))
      . $_
      . pack('N*', length($_));
   } @$ary;
}

my @o=('.1.3.4.5.7','1.2 3.4.8','3.2 9.1 7','1 3 4 5 9');

print join "\n",qw(grt:),oid_grt_sort(\@o),"";
print join "\n",qw(---- lex:),oid_lex_sort(@o),"";

HTH

[Anonymous Monk]

Thanks...it looks like the oid_lex_sort might be faster.

Thanks again!

[Limbic~Region]

Anonymous Monk,
The oid_lex_sort() function uses the Schwartzian Transform. Additionally, it uses regular expressions to eliminate leading periods and converts spaces to '.0' globally. If you know none of your OIDs require those regexes then the following might be better. Benchmarking will say for sure.

my @OID; # defined elsewhere
@OID = @OID[
    map { unpack "N", substr($_,-4) }
    sort
    map { pack('N*', split('\.', $OID[$_])) .  pack "N", $_ }
    0..$#list
];

It is a straight forward translation to tye's one true sort.
Update: Other improvements likely exist but I don't know what constitutes a lexicographical sort of OIDs.

[salva]

try using Sort::Key, it is faster and uses less memory than both the ST and the GRT:

use Sort::Key qw(keysort);

@sorted = keysort { pack "N*" => split /\./, $_ } @oids;

[Anonymous Monk]

WOW !!! Much faster, around 50% percent faster and the memory footprint was 75% less.

Thanks!

[salva]

I think that you can make it even faster and less memory hungry encoding the oid numbers as unicode chars:

@sorted = keysort { pack "U*" => split /\./, $_ } @oids;

[japhy]

What is an "OBJECT IDENTIFIER"? The docs keep stating "dotted", so is it a N.N.N(.N.N.N...) structure?

[jhourcle]

Yes. And it's of indeterminate length, as it uses delegation.

For instance:

Read more... (1499 Bytes)