Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Opening random files then reading random lines from file.

by fame (Initiate)
on Apr 27, 2012 at 04:54 UTC ( #967518=perlquestion: print w/ replies, xml ) Need Help??
fame has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I have a question, I am obviously not a pro with perl but I do understand alot.. My question is this, how can I open random files (only files which I want to be opened) example: ny.txt ca.txt nj.txt pa.txt ma.txt wa.txt

Say those are the files I want to be opened in random how would I go about doing that and then if a certain file is opened, say ma.txt was opened from the random selection I would like a random line from that file printed. What would be the proper way to do something like this?

Thanks in advance to all who respond

Comment on Opening random files then reading random lines from file.
Re: Opening random files then reading random lines from file.
by Marshall (Prior) on Apr 27, 2012 at 06:08 UTC
    Some "simple" solutions for you.
    Be aware that the Perl rand() function is not completely random - works ok for most situations but is not really "random" - there are biases in this (odd vs even), etc.

    If you want to use rand() to plan Casino strategies, there are better number generators!

    But this is one way to get the job done. I do recommend buying a copy of the Perl Cookbook.

    #!/usr/bin/perl -w use strict; #pick a "random arg from a list" my @names = qw(ny.txt ca.txt nj.txt pa.txt ma.txt wa.txt); my $random_file = $names[rand(@names)]; #UPDATE: previous code had #an "off-by-one" error, Oops! print "$random_file\n";
    Pick A Random Line from a file:
    From: Perl Cookbook
    #Chapter 8.6. Picking a Random Line from a File #Problem: You want to return a random line from a file. #Use rand and $. (the current line number) to decide #which line to print: srand; rand($.) < 1 && ($line = $_) while <>;
    Exactly how this works is complicated.
    The "Cookbook" is a great book and highly recommended by me.
      my $random_file = $names[rand(@names-1)];

      That will always skip the last element of @names.    It should be:

      my $random_file = $names[ rand @names ];
        Correct.

        These pesky "off-by-one" errors (the most common error in programming) are indeed annoying! But yes, you are correct! Thanks!

        previous post updated with comment.

      Exactly how this works is complicated.
      No, it isn't. How it works is as follows: we read a file, line by line. For the kth line read, we decide with probability 1/k whether we remember the line, forgetting any previous one.

      That's fairly trivial. It takes just a little analysis to see it's fair. Fair means that any line will be the final pick. So, what's the probability of line k to be the final pick? First what needs to be happen is that it gets remembered when the line is read, which has a probability of 1/k. Then, any next lines must *not* be picked. For each of the next lines, this happens with probability (j -1)/j, where j is the line number. Since all those probabilities are independent of each other, we can multiply them:

                 n
              -------
          1    |   |    j-1        1     k      1 
         ---   |   |    ---   ==  --- * --- == ---
          k    |   |     j         k     n      n
               j=k+1
      
      Too bad we can't really do math in HTML.
        I think you made my point that it is "complicated"
        (not easily explained and understood)..
Re: Opening random files then reading random lines from file.
by rovf (Priest) on Apr 27, 2012 at 08:21 UTC

    I see that most part of your question has already been answered. However, one more note about random line selection: Since you need to know the number of the lines in the file, you have to read it at least once. Unless the file is really big, I would therefore slurp the file into an array, and then select a line at random from this array.

    UPDATE: I was wrong. As we can see from this comment, it is indeed NOT necessary to slurp the whole file!

    -- 
    Ronald Fischer <ynnor@mm.st>
      If I correctly understand the
      rand($.) < 1 && ($line = $_) while <>
      there is no need to know the number of the lines in the file.

        Yep. This idea is beautiful.

        • First line: $.== 1 => 1/1 chance to keep this line1.
        • Second line: $. == 2 => 1/2 chance to keep line2 (so 1-(1/2)=1/2 to keep line1) (Equiprobability)
        • Third line: $. == 3 => 1/3 chance to keep line3 ( so (1-(1/3))=2/3 chance to keep a previous line (line1 or line2) : (2/3)*(1/2)=1/3 for line1 and same thing 1/3 for line2 (Equiprobability)
        • (N+1)th line: $. == N+1 => 1/(N+1) chance to keep line(N+1) so (N+1 -1)/(N+1) to keep a previous line, one of all previous N lines (each one has 1/N): for each previous line probability is ((N+1 -1)/(N+1))*(1/N) = 1/(N+1) (Equiprobability again)
        You are absolutely right; when I wrote my comment, I did not know yet the ingenious (and simple!) algorithm proposed by JavaFan.


        -- 
        Ronald Fischer <ynnor@mm.st>
      Since you need to know the number of the lines in the file,
      No, you don't. There's a fair and linear way to pick a random element from a stream of unknown length, and its algorithm (which can be implemented as a one-liner) has been displayed in a post before yours.

      Detailed analysis can be found in The Art of Computer Programming, by Donald Knuth. Volume 1, IIRC.

      yeah most of my question has been answered. but i got it down now.. thanks all for the responses.. its greatly appreciated..

      I knew how to get a random line read in a file but didnt quite understand how to get a random file opened and do what i needed until i started playing around with it and took advice of others and got a copy of perl cookbook..the following is what i came up with for what I need to be done

      #!/usr/bin/perl my @names = qw(x.txt l.txt m.txt); my $random_file = $names[rand @names]; if($random_file eq "l.txt") { open FILE, "l.txt" or die "Error Opening: Cannot open file!"; @emails = <FILE>; close FILE; $email = $emails[int(rand($#emails +1))]; chomp($email); print "$email\n"; }elsif($random_file eq "x.txt"){ open FILE, "x.txt" or die "Error Opening: Cannot open file!"; @emails = <FILE>; close FILE; $email = $emails[int(rand($#emails +1))]; chomp($email); print "$email\n"; }elsif($random_file eq "m.txt"){ open FILE, "m.txt" or die "Error Opening: Cannot open file!"; @emails = <FILE>; close FILE; $email = $emails[int(rand($#emails +1))]; chomp($email); print "$email\n"; }

      any feedback on this would also be appreciated... any suggestions or tips to better the way i am doing this? any and all information is always appreciated :) thanks again

        I read elsewhere (sorry but I forgot where) that there's one potential technical problem with this algorithm:

        If your rand(n) function isn't strictly uniform, neither will your random line selection, and this problem tends to get worse if the total number of lines increases.

        A naively implemented rand(n) is often non-uniform because your RNG outputs integers between 0 and RMAX, where RMAX is usually a power of 2, or a large prime. The quick-n-dirty way to turn this into a random number between 0 and n, is to calculate its remainder modulo n. This is only truly uniform if n exactly divides RMAX, otherwise it will be almost uniform (given that RMAX is big), which is good enough for many purposes, but not if you're doing many calls and the correctness of your algorithm depends on the uniformity of your rand(n). The proper way to do it, btw, is to take the modulo in most cases, but re-roll if the RNG output is larger than n * floor(RMAX / n).

        I am not sure how the rand(n) function in Perl works internally, hopefully it's been corrected over the years, but for other languages it's something to be on the lookout for. The only one I'm certain of is that Python's got the correct algo, because I remember reading a bug report about their random range function.

Re: Opening random files then reading random lines from file.
by BillKSmith (Chaplain) on Apr 27, 2012 at 15:26 UTC

    You say that you want to open random files (plural). If you really mean to open all the files in random order (no repeats) you have more to do than putting one of the previous suggestions in a loop. You should 'shuffle' the list of file names and then open the files in the order of the shuffled list. The "Perl Cookbook" also discusses this in some detail. I recommend the shuffle function in the module List::Util

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://967518]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (12)
As of 2014-07-30 21:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (241 votes), past polls