Random numbers are not random enough on Windows

[kikuchiyo]

I noticed something about the built-in random number generator.

I tried the following:

perl -le'$n=1000000;for(1..$n){$x=rand;$h{$x}++}@b=sort keys %h;$,="\t
+";print $h{$b[$_]},$b[$_] for 1..$n;print STDERR scalar @b' > randoms
+.txt

This code generates 1000000 random numbers, performs the equivalent of sort | uniq -c on them, prints the unique numbers (with their frequencies) into the file randoms.txt, and also prints how many unique random numbers were generated.

On linux (Perl 5.10.0 coming straight from the Ubuntu 9.04 repo) this prints 1000000, meaning that every random number generated was unique. This was expected.

However, on Windows (Perl 5.10.0, the latest Camelbox build) this prints 32768, meaning that only that many unique numbers were generated and there were many duplicates.

Why is the random number generator so crippled in that particular build? Or is this platform-specific?

[GrandFather]

Perl's rand uses the C run time rand provided by the compiler. Often such pseudo-random number generators perform very poorly and the Microsoft one provided by the compiler used to build ActiveState's Perl offering is a case in point.

However, it's a problem that is easily fixed. Just use Math::Random::MT qw(srand rand); and the problem goes away regardless of how your Perl was built.

[almut]

Yes, this a known issue.  You could use other pseudo random number generators instead, like Math::Random::MT (see Mersenne_twister for background) or Math::Random.

[jettero]

every random number generated was unique. This was expected.

I had a stats teacher that would have everyone write down a coin flip sequence and then he'd explain how he could tell who cheated: not enough repeats. I guess I'd expect to see a few repeats in my sequence of random numbers. Probably not as many as you're seeing... I think there are other problems with the windows PRNG too. Maybe win7 fixed it.

I guess I expected to find some things on CPAN for this, but I didn't -- unless you're willing to use cygwin. Then there seem to be a few choices. Crypt::Random (will this work without the entropy daemon?) and Math::Random::MT::Auto (should I be using this too?) seem fairly interesting.

UPDATE: Ahh, interesting. I didn't understand the issue at all.

[GrandFather]

The issue the OP experienced was not with the Windows PRNG, but with the PRNG built into Perl. They are quite different animals. The Perl PRNG is a 16 bit primitive PRNG that has been provided by MicroSoft's C run time for pretty much ever. It has long been recognised as a very flawed (some would say 'useless') PRNG, but much code has been written using it that would probably break if it were changed. If you have a serious need for a good PRNG for some application there are plenty of libraries around that will supply the need for various appropriate definitions of 'good'.

To a fairly large extent there is not a great deal of point in a compiler supplying a 'good' PRNG because there are many areas of compromise in the definition of good for any particular application. Anyone with a serious application for a PRNG will most likely either develop their own, or use library code developed for a similar application.

[kikuchiyo]

Thanks for the answers.

I know that one shouldn't use the built-in PRNG for any 'serious' work - this is true for most programming languages, not only Perl.

There are several modules implementing 'better' PRNGs on CPAN, furthermore, Perl has interfaces for many well-known and well-tested mathemathical libraries, each having a good library of PRNG functions (GMP, PARI, GSL among others). So if I were to do any kind of serious work that involved random numbers, for example cryptography or stat.phys. simulations, I'd use one of the above.

Still, it's sad to know that the built-in generator is _this_ useless on certain platforms.

[BrowserUk]

I know that one shouldn't use the built-in PRNG for any 'serious' work...

And that's pretty much why the MS C built-in PRNG has never been improved. If they improved it sufficiently that it was no longer glaringly obvious that it is useless, then someone, and probably many someones, would try to use it for critical purposes and then blame them when it all came unravelled.

And whilst I've never seen it written anywhere, that might also be a good reason for Perl not providing a better PRNG built-in. If it is sufficiently better to be worth while including, someone is going to assume it is sufficient for purposes for which it isn't.

That said, it would be nice to have a good--in the simulation sense rather than the crypto sense--PRNG as a part of the core. Preferably such that if enabled through a use (or pragma), and seeded with a known value, it would produce consistant sequences across platforms. It would be a considerable boon for testing purposes.

The MT would probably do that, but would forcing the users to explicitly select it for use be sufficient to prevent the blame game when people use it for the wrong purposes?

[Anonymous Monk]

Why not just go around crippling all kinds of methods on Linux in addition to Windows, just so that it is painfully obvious that no one should try to do anything with any built-ins? Clearly doing a shabby job is not the answer.

[biohisham]

Check rand and also there's this other module suggested by the documentation of the seeding function srand to generate more randomized values "Math::TrulyRandom" check it in the CPAN..

[markuhs]

Same value for me on ActiveState 5.10.0 build 1005...

32768 -> 16 bit ???

[almut]

$ perl -V:randbits

On my Ubuntu box this prints randbits='48';  On your build of Perl I suspect it's 15.

(This value is figured out at build-time and is supposed to represent how many bits the respective PRNG C library function actually provides.)

[ikegami]

ActiveState Windows build:

>perl -V:randbits
randbits='15';

[QM]

You might have also noticed that 32768 is 2**15. All PRNGs have a limit on the number of unique values they can generate.