Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Random sampling a variable record-length file.

by pajout (Curate)
on Dec 30, 2009 at 13:35 UTC ( #814938=note: print w/replies, xml ) Need Help??


in reply to Random sampling a variable record-length file.

I see following algorithm, assuming ~10G file of records of 50-150 bytes, which ordering/length can be somehow dependent:
1. Estimate count of records in result sample: $n = 1000; 2. Estimate count of records in "primar" sample: $p = $n * 100; 3. Generate array of byte positions @pos, uniformly distributed in int +erval [1, length of file], satisfying condition $p == @pos; 4. If it is worth for seeking in the file, sort @pos. 5. my @res; my %res; 6. foreach @pos 6.1 read the record starting behind $_-th byte and remember $start, wh +ich is position of the first byte of the record in the file 6.2 next if exists $res{$start}; 6.3 $res{$start} = 1; 6.4 my $rand = some_uniform_rand()/($start - $_). It is crucial step - + IF ordering of records is somehow dependent on record length, it can + normalize it. Records following longer records have lowered probabil +ity to be inserted into @res. 6.5 if $n < @res insert {weight => $rand, rec => $record} into @res, keeping asc +ending order by weight 6.6 elsif $rand > $rec[0]{weight} shift @rec and insert {weight => $rand, rec => $record} into @r +es, keeping ascending order by weight 7. you have @res
UPDATE: 6.5 if $n > @res

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://814938]
help
Chatterbox?
[ambrus]: Like <c>{ package AnyEvent::Impl:: Prima; use Prima; sub io{ my($s,%r)=@_; Prima::File->new( file=>$r{fh},mask =>("w"eq$r{poll}? fe::Write():fe:: Read())|fe:: Exception,onRead =>$r{cb},onWrite =>$r{cb}, onException=>$r{cb }) } sub timer { ... } push @AnyEvent::REGI
[ambrus]: argh, too long, let me try on scratchpad
[Corion]: . o O ( I seem to have improved my skills of getting other people to write code for me )
[ambrus]: ambrus's scratchpad

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (9)
As of 2016-12-08 12:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:













    Results (141 votes). Check out past polls.