Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Hash-type lookup in PDL

by andye (Curate)
on Jan 05, 2007 at 13:26 UTC ( #593108=perlquestion: print w/ replies, xml ) Need Help??
andye has asked for the wisdom of the Perl Monks concerning the following question:

Hi folks,

I'm using PDL (the Perl Data Language, pdl.perl.org) to manipulate a Very Large dataset.

I'm having quite a bit of trouble with the correct syntax for threading, can't seem to get it right.

Right now I'm doing a lookup, where an old value gets turned into a new value. There's a one-to-one relationship between old values and new values. Currently I'm doing this using a normal hash, like this:

my @old_values = list($pdl_oldvalues); my @new_values = map $hash_lookup->{$_}, @old_values; my $pdl_newvalues = pdl(\@new_values);

I'd like to do this using PDL threading if I can, because I want it to go quicker (there are a lot of values), but I can't find the right way to do it.

I've tried:

my $pdl_newvals = $hash_lookup->{$pdl_oldvals};
but sadly that doesn't work. I've tried using $pdl_oldvals->thread(0) and $pdl_oldvals->dummy(0) but they don't work either.

I've also tried turning the hash into a two-row pdl, with keys on one row and values on the other row, then taking a slice from that pdl to get the values. That should certainly work with pdl threading... but I can't get it to. :(

Suggestions, and examples of correct syntax for doing this kind of thing in pdl, very much appreciated.

All the best, andye.

PS where "doesn't work" = either a syntax error, or runs but produces the wrong values...

Comment on Hash-type lookup in PDL
Select or Download Code
Re: Hash-type lookup in PDL
by zentara (Archbishop) on Jan 05, 2007 at 17:28 UTC
    Could you possibly work up a simple running script, with sample data, so we can see what format your data is in, and the dimensions of your piddles? It sounds to me my you are looking for "slices", to do the changes in place (efficiently), but I have no idea what your piddle looks like.

    The cool thing about slices, is that whatever operation you perform on the slice, also occurs to the original piddle.

    #!/usr/bin/perl use warnings; use strict; use PDL::LiteF; my $x = sequence(25); # Create a piddle of increasing value print "$x\n"; my $slice = slice $x,'4:20:2'; print $slice; print "\n"; $slice .= 0; print $slice; print "\n"; print $x; print "\n"; for (1..100){ $slice ++; print $x; print "\n"; select(undef,undef,undef,.5); }

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
      Hi zentara, It's nothing complicated I'm looking to do, really, just lookup each value in a hash. Here's an example that doesn't work:
      #!/usr/bin/perl use warnings; use strict; use PDL; my $x = sequence(5); print "$x \n"; my $lookup = { 0 => 101, 1 => 69, 2 => 42, 3 => 10042, 4 => 99 }; my $y = zeroes(5); $y .= $lookup->{$x}; print "$y \n";
      That gives this output:
      [0 1 2 3 4] [0 0 0 0 0]
      I would be a happy bunny if it gave this output:
      [0 1 2 3 4] [101 69 42 10042 99]
      Best wishes, andye

      (I tried using a straightforward my $y = $lookup->{$x} but that doesn't produce any value at all)

      update: I've tried creating a pdl to use insread of the hash, like this:

      my $pdl_table = pdl([keys %$lookup], [values %$lookup]); my $y = zeroes(5); $y .= $pdl_table->(which($pdl_table->(:,0) == $x),1 +); print "$y \n";
      but that has two problems:
      • It's rather hard to read
      • It doesn't work
      Looks like I've screwed up the syntax in the which() statement, and I can't work out how to make it right - a dummy dimension somewhere?

        Hi andye,

        If your lookup table is rather small, you could try this code:

        #!/usr/bin/perl use warnings; use strict; use PDL; my $x = sequence(5); print "$x \n"; my @pdl_keys = (0, 1, 2, 3, 4); my @pdl_values = (101, 69, 42, 10042, 99); my $y = zeroes(5); foreach my $i (0..@pdl_keys-1){ my $indx = which($x == $pdl_keys[$i]); $y->dice($indx) .= $pdl_values[$i]; } print "$y \n";

        I hope this helps

        Cheers,

        lin0
Re: Hash-type lookup in PDL
by zentara (Archbishop) on Jan 05, 2007 at 22:03 UTC
    Here are a couple of other ideas for you.
    #!/usr/bin/perl use warnings; use strict; use PDL; my $x = sequence(5); print "$x \n"; my $lookup = { 0 => 101, 1 => 69, 2 => 42, 3 => 10042, 4 => 99 }; my $y = zeroes(5); foreach (keys %{$lookup}){ set $y, ,$_ ,$lookup->{$_}; } print "$y \n";

    This one is probably more useful, because you can read in a large amount of data.

    #!/usr/bin/perl use warnings; use strict; use PDL; use PDL::IO::Misc; my $z = rcols *DATA; print "$z\n"; print $z->at(1); print "\n"; __DATA__ 101 69 42 10042 99

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://593108]
Approved by wfsp
Front-paged by ikegami
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2014-10-25 01:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (139 votes), past polls