Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re^3: Processing values of a piddle (PDL) speedup using 'at' vs. 'index'

by chm (Novice)
on Mar 24, 2013 at 22:23 UTC ( #1025190=note: print w/replies, xml ) Need Help??

in reply to Re^2: Processing values of a piddle (PDL) speedup using 'at' vs. 'index'
in thread Processing values of a piddle (PDL) speedup using 'at' vs. 'index'

First off: the best place to ask questions about PDL is on the perldl mailing list and the central site for info on all things PDL is

Second, you are using the right strategy here. The key to remember is that calculation with PDL objects (called "piddles") are performed with special C code and are very fast. If your work can be done on the piddle data directly, you will almost always see the best performance.

In this case, I would suggest using PDL operations to find all the "special values" in the piddle, mark them as BAD and the list() method (or the newer unpdl() method) will convert the piddle back to a perl list or list of list structure with the the special values all now having the value 'BAD'.

A map can be used to substituted undef if that is needed for your algorithm. NOTE: if you don't need the special value elements at all, it is easy to not include them in the list() output via PDL operations.

Here is a short session with the PDL shell (pdl2) showing some calculations along these lines:

pdl> apropos bad # PDL shells have online help PDL::Bad Module: PDL does process bad values PDL::BadValues Manual: Discussion of bad value support badflag getter/setter for the bad data flag badinfo information on the bad-value support ...many more... pdl> help isbad Module PDL::Bad isbad Signature: (a(); int [o]b()) Returns a binary mask indicating which values of the input are bad values Returns a 1 if the value is bad, 0 otherwise. Similar to isfinite. $a = pdl(1,2,3); $a->badflag(1); set($a,1,$a->badvalue); $b = isbad($a); print $b, "\n"; [0 1 0] This method works with input piddles that are bad. The output piddle will never contain bad values, but its bad value flag will be the same as the input piddle's flag. pdl> $data = rint(10*random(10)) pdl> p $data [5 9 8 3 5 6 7 7 6 10] pdl> $special = 7 pdl> p $data->setvaltobad($special) [5 9 8 3 5 6 BAD BAD 6 10] pdl> p $data->setvaltobad($special)->list 5 9 8 3 5 6 BAD BAD 6 10 pdl> @pdata = $data->setvaltobad($special)->list pdl> p "@pdata" 5 9 8 3 5 6 BAD BAD 6 10 pdl> foreach (@pdata) { $_ = undef if $_ eq 'BAD' } pdl> p "@pdata" Use of uninitialized value $pdata[6] ... Use of uninitialized value $pdata[7] ... 5 9 8 3 5 6 6 10 pdl> p which $data==$special # calc indices of "special vals" [6 7] pdl> @ordinary = $data->where($data != $special) pdl> p "@ordinary" # or output just ordinary values [5 9 8 3 5 6 6 10]
  • Comment on Re^3: Processing values of a piddle (PDL) speedup using 'at' vs. 'index'
  • Download Code

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1025190]
[Corion]: Meh. I have a very simple mechanism to scrape+track prices, and even to send me mail if something changes (via cron), but SQLite doesn't support window functions, so my simple SQL to determine a price change won't work :-( Maybe I should store the DB ...
[Corion]: ... in Pg, but that would mean that I'd have to deal with credentials and stuff :)
[Corion]: In fact, the whole thing is just four simple programs, one App::scrape, another being DBIx::RunSQL, and one some glue to convert a JSON object into an SQL INSERT statement (+DBI connect/execute), so it would be an incredibly simple solution...
[Corion]: ... but that simplicity falls down due to SQLite letting me down and me being too lazy to move to a real DB. Maybe DBD::CSV can handle window functions...
[hippo]: Pg has the "trust" mechanism if you don't fancy using credentials for a trivial db.
[Corion]: hippo: Oooh, that sounds quite nice - let me read the documentation on that :)

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2018-02-23 15:05 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (302 votes). Check out past polls.