in reply to Re: Processing values of a piddle (PDL) speedup using 'at' vs. 'index'
in thread Processing values of a piddle (PDL) speedup using 'at' vs. 'index'
Thanks for the reply.
I think your comment,
My (very limited) experience of working with PDL suggests that if you need to manipulate the values in a piddle individually, rather than applying each operation to the entire piddle as a whole, then you should export the piddle, en-masse, to a perl array first. It saves huge amounts of time,
and this excerpt from the PDL Book,
Be careful with at, as you almost never want to use it - it is tedious for anything nontrivial, and extremely slow! Particularly if you find yourself placing an at call inside a for loop, you should probably stop and think about how to use threading for your problem - see below.
are getting at the same idea. That is, one should avoid processing values in a piddle individually and take advantage of PDL's various commands for manipulating whole vecotrs or matrices.
The code example that I gave is a bit simplified compared to my actual use case. In my case, I iterate through a few thousand objects (these objects have attributes that are 1-dimensional pdls). The values from most (or all) of these pdls need to be exported into my text file. I take values from these pdls, check them to see if they are a special value, change the value if needed, and then put them into a perl array. The perl array is eventually written to a text file. Your comments have me thinking that another possibility might be to do something like this:
It's possible that this type of approach might be faster; however, the current approach using at is working plenty fast for me at the moment (other parts of my code are now the bottleneck).
- create an n-dimensional pdl (i.e. a pdl of 1-dimensional pdls)
- then transform and/or take slices of the n-dimensional pdl for output to my text file (using PDL commands)
- replace the special values (using a PDL commands...I am unsure which ones would apply here)
- create the text file