Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: File Processing...

by ayrnieu (Beadle)
on Mar 20, 2006 at 00:08 UTC ( #537837=note: print w/replies, xml ) Need Help??

in reply to File Processing...

use Tie::File; I have seen it quite capable of dealing with very large files given very little wiggle room.

Oh, wait, 200MBs? Nevermind.

It doesn't seem that you need exactly 200MBs, as you want to know how to conditionally go over 200MBs; also, you are dealing with text files... is it possible that you actually want to split the file by your 'certain string'?

Replies are listed 'Best First'.
Re^2: File Processing...
by Anonymous Monk on Mar 20, 2006 at 01:55 UTC
    Yeah, it will be +/- 200mbs.

    I asked them this too, if I could just split the file by strings...that'd be easy....

    Since I don't know what the data is for or from(they don't tell me, since I am not a permanent employee) they just said that these strings are too frequent among the file. They're something like START_HTML and END_HMTL is what I've heard. Maybe its a web crawler?

    But they said that splitting based on this would make thousands and thousands of files. So I can't.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://537837]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2022-01-22 13:01 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (62 votes). Check out past polls.