Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Edit huge file

by frozenwithjoy (Curate)
on Jun 18, 2013 at 22:18 UTC ( #1039678=note: print w/ replies, xml ) Need Help??


in reply to Edit huge file

First thing that pops into mind for me if you really want to avoid reading/re-writing entire file:

  • Read in file until you find line near start you want to delete (note line number using $.
  • Read in file in reverse until you find line near end to delete (can use File::ReadBackwards)
  • Use Tie::File to represent file as an array
  • Use splice to get rid of unwanted lines
  • Then untie @array;

Disclaimer: I have limited experience with Tie::File, so not sure how it performs w/ larger files.


Comment on Re: Edit huge file
Select or Download Code
Re^2: Edit huge file
by BrowserUk (Pope) on Jun 18, 2013 at 22:45 UTC
    Disclaimer: I have limited experience with Tie::File, so not sure how it performs w/ larger files.

    Horribly!


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks. For my own future reference, do you have a suggestion for a file size that is too large to use with Tie::File?

        Hm. Say 1000 lines/100kB.

        Its actually not too bad for files ~10x that iff you know

        1. how it works:

          An LRU cache of limited size:

        2. what not to do:

          Don't use @theArray in either a list or scalar context;

        3. configure it correctly:

          The memory and dw_size options can interact in interesting but unintuitive ways.

          Bigger is not always better, especially if the predominant operations are deletion or insertion of whole lines, rather than in-place modification of existing lines.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1039678]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-08-29 06:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (275 votes), past polls