Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
I'm on CentOS, a Linux disto, not Windows.

Sorry. I remembered this reference to XP, but misremembered who made it.

I did issue the 'sync' command, but it apparently only commits buffer cache to disk rather than emptying it.

A crude but usually effective way of flushing one file from the cache is to cat a file that is bigger than the cache. Say, copy/append your 80GB datafile to another file 5 times (=400GB), and then cat that to /dev/null before running your tests. Might work for you.

I'm able to cat my testfile to /dev/null in under one second.

Assuming this is your 10e6 record testfile, and it is representative of your 80GB file and has an average of 86 characters/line, that gives a filesize of ~820MB.

The very best sequential-read throughput figure I can find for a non-raided 15k local drive is a little over 100MB/s.

That pretty much confirms that your testing is reading from cache rather than from disk. Even the most optimistic read-ahead algorithm cannot drive the interface 8 times faster than its maximum throughput.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^13: selecting columns from a tab-separated-values file by BrowserUk
in thread selecting columns from a tab-separated-values file by ibm1620

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others studying the Monastery: (10)
    As of 2015-07-30 07:00 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









      Results (270 votes), past polls