Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^3: Displaying/buffering huge text files

by JavaFan (Canon)
on Dec 23, 2011 at 10:32 UTC ( #944904=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Displaying/buffering huge text files
in thread Displaying/buffering huge text files

I, for one, would like to see your code for building an 5-byte index?
Storing a million 5-byte integers using 5 million bytes for data, and 44 bytes additional overhead:
my $buffer = ""; my $BYTES = 5; my $BITS_IN_BYTE = 8; my $FULL_BYTE = (1 << $BITS_IN_BYTE) - 1; sub store { my ($pos, $value) = @_; for (0 .. $BYTES - 1) { vec($buffer, $pos * $BYTES + $_, $BITS_IN_BYTE) = ($value >> ($BITS_IN_BYTE * ($BYTES - 1 - $_))) & $FULL_BYT +E; } } sub fetch { my ($pos) = @_; my $sum = 0; for (0 .. $BYTES - 1) { $sum <<= $BITS_IN_BYTE; $sum += vec($buffer, $pos * $BYTES + $_, $BITS_IN_BYTE); } $sum; } # # Testing # my $TEST_SIZE = 1_000_000; my @offsets = map {int rand 1 << ($BYTES * $BITS_IN_BYTE)} 1 .. $TEST_ +SIZE; for (my $i = 0; $i < @offsets; $i++) { store $i, $offsets[$i]; } for (my $i = 0; $i < @offsets; $i++) { my $val = fetch $i; die unless $val == $offsets[$i]; } use Devel::Size 'size'; use 5.010; say "Index size: ", size $buffer; __END__ Index size: 5000044


Comment on Re^3: Displaying/buffering huge text files
Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://944904]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (6)
As of 2014-09-18 09:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (109 votes), past polls