Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Re: selecting N random lines from a file in one pass

by boo_radley (Parson)
on Dec 22, 2004 at 17:02 UTC ( #416825=note: print w/replies, xml ) Need Help??

in reply to selecting N random lines from a file in one pass

behold my shameless self promotion.
  • Comment on Re: selecting N random lines from a file in one pass

Replies are listed 'Best First'.
Re^2: selecting N random lines from a file in one pass
by seuratt (Novice) on Dec 22, 2004 at 17:39 UTC
    Your self-promotion requires reading the entire file into memory before it can be passed to the module, which makes it kind of inapplicable to problem.

    However, if you do have the entire data set in memory, there's a simpler approach than the math theory you use in your module. Rather than iterate the array, consider just drawing at random. If there's a duplicate, then redraw. The larger the set, the less the odds of a duplicate. This scales a lot better than your module, but I admit that does trip up more on sets close to sample size.

    It's a lot like how, in gaming, you can fake a d12 by rolling a d20 until you've got a number in range.

    use strict; use Data::Dumper; my @set = ('a'..'z'); my $size = 3; my @sample; my %seen; while( @sample < $size ){ my $elt; do{ $elt = int rand(@set); }while( $seen{$elt} ); push @sample, $set[$elt]; $seen{$elt}++; } print Dumper \@sample;

    This example should really have a check, too, to see if the sample size is in the bounds of the set, to avoid looping.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://416825]
[Corion]: Meh. My bank removed (part of) their CSV download facilities. Now I will either have to implement a full scraper or automate the download using the HBCI interface instead (or just get a new account elsewhere...)
[Corion]: On the upside, I spend a lot of time thinking this weekend about how to actually implement rate limiting for futures, and if things work out, maybe even loading a configuration from an external file makes sense
[Corion]: I've also found some interesting invariants that I have to think/write about more. A simple rate limiter will never change the order of the input, while a limiter that allows for parallel execution will change the order. But my API currently allows for bo
[Corion]: ... for both, and I'm not sure if I want to add the cruft from the parallel API (a token that you need to hold on to while you hold the lock) to the rate limiting API too, to allow seamless up/downgrades, or not.
[Corion]: Also, rate limiting will look great with await: my $token = await $limiter-> limit($hostname); instead of my $f = $limiter->limit( $hostname )->then(sub { my( $token)=@_; ... });

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (9)
As of 2017-10-23 08:23 GMT
Find Nodes?
    Voting Booth?
    My fridge is mostly full of:

    Results (277 votes). Check out past polls.