Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Re: Need to process a tab delimted file *FAST*

by clintp (Curate)
on Mar 03, 2004 at 13:56 UTC ( #333540=note: print w/replies, xml ) Need Help??


in reply to Re: Need to process a tab delimted file *FAST*
in thread Need to process a tab delimted file *FAST*

If the I/O isn't your problem then consider the following:

* Don't use Perl. Re-write this in C, keep your keys in a tree of some kind (determined by the distribution of the keys) and the values (tot, max, last) in the tree. It's always faster to use a specifically designed tool (a single-purpose, well written C program) than a general purpose multitool (like Perl).

* If you have any control over the input stream: instead of tab-delimited data, use space/column-delimited data. It's (probably) faster to extract two substrings of known width than to split. Leave the whitespace on the keys when storing, remove it later when redisplaying (if necessary).

  • Comment on Re: Re: Need to process a tab delimted file *FAST*

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://333540]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (10)
As of 2016-12-07 14:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:













    Results (127 votes). Check out past polls.