Re: Opening random files then reading random lines from file.

[rovf]

in reply to Opening random files then reading random lines from file.

I see that most part of your question has already been answered. However, one more note about random line selection: Since you need to know the number of the lines in the file, you have to read it at least once. Unless the file is really big, I would therefore slurp the file into an array, and then select a line at random from this array.

UPDATE: I was wrong. As we can see from this comment, it is indeed NOT necessary to slurp the whole file!

[choroba]

If I correctly understand the

rand($.) < 1 && ($line = $_) while <>

there is no need to know the number of the lines in the file.

[brx]

Yep. This idea is beautiful.

• First line: $.== 1 => 1/1 chance to keep this line1.
• Second line: $. == 2 => 1/2 chance to keep line2 (so 1-(1/2)=1/2 to keep line1) (Equiprobability)
• Third line: $. == 3 => 1/3 chance to keep line3 ( so (1-(1/3))=2/3 chance to keep a previous line (line1 or line2) : (2/3)*(1/2)=1/3 for line1 and same thing 1/3 for line2 (Equiprobability)
• (N+1)th line: $. == N+1 => 1/(N+1) chance to keep line(N+1) so (N+1 -1)/(N+1) to keep a previous line, one of all previous N lines (each one has 1/N): for each previous line probability is ((N+1 -1)/(N+1))*(1/N) = 1/(N+1) (Equiprobability again)

[rovf]

You are absolutely right; when I wrote my comment, I did not know yet the ingenious (and simple!) algorithm proposed by JavaFan.

[JavaFan]

Since you need to know the number of the lines in the file,

No, you don't. There's a fair and linear way to pick a random element from a stream of unknown length, and its algorithm (which can be implemented as a one-liner) has been displayed in a post before yours.

Detailed analysis can be found in The Art of Computer Programming, by Donald Knuth. Volume 1, IIRC.

[fame]

yeah most of my question has been answered. but i got it down now.. thanks all for the responses.. its greatly appreciated..

I knew how to get a random line read in a file but didnt quite understand how to get a random file opened and do what i needed until i started playing around with it and took advice of others and got a copy of perl cookbook..the following is what i came up with for what I need to be done

#!/usr/bin/perl

my @names = qw(x.txt l.txt m.txt);
my $random_file = $names[rand @names];

if($random_file eq "l.txt") {

open FILE, "l.txt" or die "Error Opening: Cannot open file!";
@emails = <FILE>;
close FILE;
$email = $emails[int(rand($#emails +1))];
chomp($email);
print "$email\n";

}elsif($random_file eq "x.txt"){

open FILE, "x.txt" or die "Error Opening: Cannot open file!";
@emails = <FILE>;
close FILE;
$email = $emails[int(rand($#emails +1))];
chomp($email);
print "$email\n";

}elsif($random_file eq "m.txt"){

open FILE, "m.txt" or die "Error Opening: Cannot open file!";
@emails = <FILE>;
close FILE;
$email = $emails[int(rand($#emails +1))];
chomp($email);
print "$email\n";
}

any feedback on this would also be appreciated... any suggestions or tips to better the way i am doing this? any and all information is always appreciated :) thanks again

[Anonymous Monk]

I read elsewhere (sorry but I forgot where) that there's one potential technical problem with this algorithm:

If your rand(n) function isn't strictly uniform, neither will your random line selection, and this problem tends to get worse if the total number of lines increases.

A naively implemented rand(n) is often non-uniform because your RNG outputs integers between 0 and RMAX, where RMAX is usually a power of 2, or a large prime. The quick-n-dirty way to turn this into a random number between 0 and n, is to calculate its remainder modulo n. This is only truly uniform if n exactly divides RMAX, otherwise it will be almost uniform (given that RMAX is big), which is good enough for many purposes, but not if you're doing many calls and the correctness of your algorithm depends on the uniformity of your rand(n). The proper way to do it, btw, is to take the modulo in most cases, but re-roll if the RNG output is larger than n * floor(RMAX / n).

I am not sure how the rand(n) function in Perl works internally, hopefully it's been corrected over the years, but for other languages it's something to be on the lookout for. The only one I'm certain of is that Python's got the correct algo, because I remember reading a bug report about their random range function.

[jdporter]

Or use the Mersenne twister. Search cpan.