Simple LWP Exercise

[bilfurd]

Somebody on dilbert.com (might have been Scott Adams) was looking for an application that would set a browser to cycle through a list of sites until the user clicked on the page to read it. This bugged me off and on for over a year.

The logic is simple. The easiest way would be using JavaScript and frames. Of course, I have better things to do with my time than staring at a monitor waiting to click when somebody posts new content. I wanted the computer to do the work and let me know when a page changed.

Between redsquirrel's recent Auto-Surfer, an old post from kilinrax, and The Mouse Book, I put together the following:

#!/usr/bin/perl -w

use strict;
use LWP::UserAgent;

my $pageNew;

# I used the US version of http://www.theregister.co.uk/ as an example
my $siteName = 'http://www.theregus.com'; 

my $ua = LWP::UserAgent->new;

my $request = HTTP::Request->new('GET', "$siteName");
my $pageOld = $ua->request($request);

while() {
   
   sleep 1800; # 30 Minutes
   
   $pageNew = $ua->request($request);
   
   if ($pageOld->content ne $pageNew->content) {
      system('konqueror', $siteName)==0 
         or die "Unable to open browser\n";
         
      $pageOld = $pageNew;
   }         
}

I know what you are thinking: 'Hooray! Good for ol' Bilfurd!' and possibly 'Get the net, another loony found a keyboard!' You forgot my favorite - 'Here's proof that the Mac made using a PC so easy even an idiot can do it.'

I actually found a use for this involving a luser (I took his keyboard away once already...) and an intranet page showing web statistics. I have a few questions before I install it on his PC:

• How can this be improved?
• What else can this be used for?
• Has anyone had success using electric shock to train users?

It should be interesting to see what the monks' collective wisdom yields on this one...

[Dog and Pony]

I am using something similar for myself, to try and find a new apartment. There is this site for my town that posts available apartments, and you can then apply for them at this site. Apartments are extremely rare here in Gothenburg, and usually the one that gets first application (with good enough credentials) gets the apartment. Given this, and the fact that new entries are not even every day sometimes, you would need to sit and refresh your browser 24/7 to even have a shot at it without extreme luck.

Anyhow, my script runs in the background and polls the page at regular intervals (with my preferred search options) for changes, and if there are any, it fires up a browser with the page in question so I can see if it seems interesting and then apply. Just like yours do.

I have this running here on my comp while I am working.

I have one "significant" extra, and that is a few regexps that strips everything but the apt list from the HTML before comparing, due ot the fact that otherwise dynamics like different ads or such could affect the size comparison. This is probably not necessary for many pages, not even sure if it is for the one I am polling. :)

What all those words are trying to say is that yes, there is a use for such a script. :)

As for improvements, well, I guess it could be more configurable - maybe even put it in a module on its own, so you could just say something like:

check_site( url => $url, 
            interval => 1800,
            process => \&clean_page,
            etc => 'more options',
);

Well, you get the idea. In that case, I guess it should also fork/thread, so you could poll several sites at the same time.

Not if I know if there would be a great demand for such a module, but hey - if nothing else, it may be fun to write. :)

There are rumours that cattle prods may have at least a limited effect on the behaviour of users... :)

---

You have moved into a dark place.
It is pitch black. You are likely to be eaten by a grue.