Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: Re: Need to process a tab delimted file *FAST*

by clintp (Curate)
on Mar 03, 2004 at 13:56 UTC ( #333540=note: print w/replies, xml ) Need Help??

in reply to Re: Need to process a tab delimted file *FAST*
in thread Need to process a tab delimted file *FAST*

If the I/O isn't your problem then consider the following:

* Don't use Perl. Re-write this in C, keep your keys in a tree of some kind (determined by the distribution of the keys) and the values (tot, max, last) in the tree. It's always faster to use a specifically designed tool (a single-purpose, well written C program) than a general purpose multitool (like Perl).

* If you have any control over the input stream: instead of tab-delimited data, use space/column-delimited data. It's (probably) faster to extract two substrings of known width than to split. Leave the whitespace on the keys when storing, remove it later when redisplaying (if necessary).

  • Comment on Re: Re: Need to process a tab delimted file *FAST*

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://333540]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2016-10-22 00:27 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (291 votes). Check out past polls.