Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Edit huge file

by frozenwithjoy (Priest)
on Jun 18, 2013 at 22:18 UTC ( #1039678=note: print w/replies, xml ) Need Help??


in reply to Edit huge file

First thing that pops into mind for me if you really want to avoid reading/re-writing entire file:

  • Read in file until you find line near start you want to delete (note line number using $.
  • Read in file in reverse until you find line near end to delete (can use File::ReadBackwards)
  • Use Tie::File to represent file as an array
  • Use splice to get rid of unwanted lines
  • Then untie @array;

Disclaimer: I have limited experience with Tie::File, so not sure how it performs w/ larger files.

Replies are listed 'Best First'.
Re^2: Edit huge file
by BrowserUk (Pope) on Jun 18, 2013 at 22:45 UTC
    Disclaimer: I have limited experience with Tie::File, so not sure how it performs w/ larger files.

    Horribly!


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks. For my own future reference, do you have a suggestion for a file size that is too large to use with Tie::File?

        Hm. Say 1000 lines/100kB.

        Its actually not too bad for files ~10x that iff you know

        1. how it works:

          An LRU cache of limited size:

        2. what not to do:

          Don't use @theArray in either a list or scalar context;

        3. configure it correctly:

          The memory and dw_size options can interact in interesting but unintuitive ways.

          Bigger is not always better, especially if the predominant operations are deletion or insertion of whole lines, rather than in-place modification of existing lines.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1039678]
help
Chatterbox?
[marto]: LanX++ asking the right questions :)
[Corion]: And I also have to talk to the technician implementing the export side of the change that this is not about them, and that maybe they should hold on to that changeset just in case the project comes back with a formally correct approach...
[Corion]: Naah - I think the "fasta" format should be familiar by now to the casual visitor of this site ;)
[Corion]: The rest would benefit from de-jargonizing. Maybe somebody versed in both, Perl and bioinformatics could write up a translation dictionary ;)
[marto]: I helped with some corrections to the Glasgow Uni Perl for Bioinformatics course, but didn't get time to do a full review. I've also patched some trivial problems with the bioperl based upon a question someone asked here.
[marto]: but I've never done any bioinformatics work, so don't know what people are talking about :P
[marto]: Corion biohisham & BioLion seem long gone
[marto]: IIRC BrowserUK understands this stuff

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (4)
As of 2017-08-17 09:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Who is your favorite scientist and why?



























    Results (285 votes). Check out past polls.

    Notices?