Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re: Re: Need to process a tab delimted file *FAST*

by clintp (Curate)
on Mar 03, 2004 at 13:56 UTC ( #333540=note: print w/ replies, xml ) Need Help??

in reply to Re: Need to process a tab delimted file *FAST*
in thread Need to process a tab delimted file *FAST*

If the I/O isn't your problem then consider the following:

* Don't use Perl. Re-write this in C, keep your keys in a tree of some kind (determined by the distribution of the keys) and the values (tot, max, last) in the tree. It's always faster to use a specifically designed tool (a single-purpose, well written C program) than a general purpose multitool (like Perl).

* If you have any control over the input stream: instead of tab-delimited data, use space/column-delimited data. It's (probably) faster to extract two substrings of known width than to split. Leave the whitespace on the keys when storing, remove it later when redisplaying (if necessary).

Comment on Re: Re: Need to process a tab delimted file *FAST*

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://333540]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2016-05-27 23:39 GMT
Find Nodes?
    Voting Booth?