The test suite was released on CPAN today, where it can be found under author id H/HM/HMBRAND as Test-Smoke-1.06.tgz.
Subscribing to smokers at perl.org is a good way to keep informed of "upgrades, malformed reports, warnings, expectations," and so on.
It's pretty easy to start smoking. You will need to create an empty directory where the perl source tree will be placed. Any files in that directory which are not part of the source tree will be removed when you run the test, so it is very important not to mix things up! On my system I just created a new user for the purpose and created a test subdirectory.
You will need to put the three scripts which are used somewhere, then edit the configuration as needed. The readme file contains specific information.
The smoke.sh script will run the test. If you set up a cronjob to run this daily this shouldn't take any more effort on your part than setiathome.
The test will do the following:
To run the tests you will need sufficient disk space to hold the perl source tree, a link fast enough to get source updates, and a computer that can handle lots of compiling. Also, you currently cannot use this test suite with windows, although that is being worked on.