Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Re: Need to process a tab delimted file *FAST*

by clintp (Curate)
on Mar 03, 2004 at 13:56 UTC ( #333540=note: print w/ replies, xml ) Need Help??


in reply to Re: Need to process a tab delimted file *FAST*
in thread Need to process a tab delimted file *FAST*

If the I/O isn't your problem then consider the following:

* Don't use Perl. Re-write this in C, keep your keys in a tree of some kind (determined by the distribution of the keys) and the values (tot, max, last) in the tree. It's always faster to use a specifically designed tool (a single-purpose, well written C program) than a general purpose multitool (like Perl).

* If you have any control over the input stream: instead of tab-delimited data, use space/column-delimited data. It's (probably) faster to extract two substrings of known width than to split. Leave the whitespace on the keys when storing, remove it later when redisplaying (if necessary).


Comment on Re: Re: Need to process a tab delimted file *FAST*

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://333540]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (9)
As of 2014-10-02 12:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (56 votes), past polls