Beefy Boxes and Bandwidth Generously Provided by pair Networks httptech
P is for Practical
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

If you are comparing two files for common/different keys, and if both files are about the same (huge) size, I guess you will have to get smarter than keeping all the information in memory (because you don't have enough memory).

If you can make an educated guess as to where in a file a key is likely to be found, you could use seek to look for the key in the file. This is horribly slow, but likely still faster than swapping memory. If you want to be fancy, you can cache parts of the file in memory.

If you cannot make an educated guess, I guess it will pay off to convert at least one file into a file with all your keys in fixed width, sorted by the keys. Then you can easily make an educated guess to find a given key. If you convert both files to that structure, you can easily find the keys missing in one of the two files by reading through the sorted key files line by line. This approach will roughly double your disk requirements.


In reply to Re: Reduce CPU utilization time in reading file using perl by Corion
in thread Reduce CPU utilization time in reading file using perl by madtoperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (7)
    As of 2014-04-19 03:47 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (477 votes), past polls