Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Re: processing large files

by Tuna (Friar)
on Jul 04, 2001 at 05:18 UTC ( [id://93746]=note: print w/replies, xml ) Need Help??

in reply to processing large files

As a network analyst for a tier-1 ISP, I can happily report that Perl can, and on a daily basis, does, process files in the neighborhood of several hundred gig. Are you sure that your limitation is not hardware-related?

Replies are listed 'Best First'.
Re: Re: processing large files
by Anonymous Monk on Jul 07, 2001 at 01:12 UTC
    How could it be hardware related if these files are on
    disk at that size and C++ programs process them?
    how do you process your files with perl? Is it a specfic
    perl? or compile option? or algorithm?

      Easily! However, you never said that you had other programs that were successfully processing your data. =) As far as my processing is concerned, I churn through Cisco NetFlow data, along with bgp tables, route summaries, snmp data, merge it all together and spit it back out again to create traffice matrices. The algorithms aren't that complicated at all; it's just a shite-load of data that I process continuously. If I tried to run on anything less than the E450 that it runs on, I would be in trouble.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://93746]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (6)
As of 2024-05-22 15:31 GMT
Find Nodes?
    Voting Booth?

    No recent polls found