Storing Experience for Posterity

[GotToBTru]

I put the following together to scrape experience, level and writeups off my profile page and store it in a text file on my computer, to record my progress through the Monastery. A scheduled task runs this once a day.

use strict;
use warnings;
use LWP::Simple;
use URI::URL;

my $date=`ECHO %DATE:~10,4%%DATE:~4,2%%DATE:~7,2%`; # YYYYMMDD

my $url = url('http://perlmonks.org/?node_id=844862');

my $content = get($url);
$content =~ s/\cJ//g;
$content =~ s/\cM//g;

my ($experience, $level, $posts) =
   ($content =~ /Experience:\D+(\d+).+
                 Level:.+([A-Z][a-z]+\s+\(\d+\)).+
                 Writeups:.+>(\d+)</x);

open my $ofh, '>>','perl_xp.dat';
printf $ofh "%d,%d,%d,%s\n",$date,$experience,$posts,$level;
close($ofh);

There is probably a way to do this in Javascript that could be included in the Free Nodelet, but that's beyond my skill level.

Update

Improved version here.

[choroba]

You might also find Average Node XP per Month interesting.

[oiskuu]

By the way, there's a perfect back-end for this sort of data: rrdtool. You might want to explore that option.

[morgon]

I would prefer to generate the timestamp in Perl rather than using backticks. One possibilty (among many):

my $date = POSIX::strftime "%Y%m%d", localtime;

And what you totally disregard is error-handling.

Your get could fail (e.g. no internet-connection), your parsing of the page could fail (the website changed it's layout) and your open could fail (some permission problem maybe).

You should handle these conditions and report appropriate errors, rather than fail silently and maybe loose the history of your meteoric raise...

[GotToBTru]

Changes:

• Added error message if web page get fails
• Use strftime instead of backtics and Windows date symbols
• Test to see if all 3 values were found before logging
• Using Tie::File for easy read-write access to file
• Replace latest line with new data if Level has not changed
• Added error message if regex could not find all 3 values
• Replace 2 s/// with 1 tr///
• Add math and reporting of anticipated level up

use strict;
use warnings;
use LWP::Simple;
use URI::URL;
use POSIX qw/strftime/;
use Tie::File;
use DateTime;

my $date= strftime "%Y%m%d", localtime; # YYYYMMDD
my ($url, $content, $experience, $level, $posts);

$url = url('http://perlmonks.org/?node_id=844862');

$content = get($url) or die "Get failure";
$content =~ tr/\cJ\cM//d;

($experience, $level, $posts) =
        ($content =~ /Experience:\D+(\d+).+
                      Level:.+([A-Z][a-z]+\s+\(\d+\)).+
                      Writeups:.+>(\d+)</x);

die unless ($experience && $level && $posts);
my ($start, $startdt, $startexp, $today);
tie my @log, 'Tie::File', 'perl_xp.csv';
($start, $startexp) = (split ',', $log[-2])[0,1];
if ($level eq (split ',',$log[-1])[3] || '') {
    pop @log;
} 
push @log, join ',',$date,$experience,$posts,$level;
untie @log;

my ($year,$month,$day) = $start =~ m/(\d{4})(\d\d)(\d\d)/;
$startdt = DateTime->new(year => $year, month=> $month, day => $day);

($year,$month,$day) = $date =~ m/(\d{4})(\d\d)(\d\d)/;
$today = DateTime->new(year => $year, month=> $month, day => $day);
my $timeflies = $startdt->delta_days($today)->{days};
my $exprate = ($experience-$startexp)/$timeflies;
printf "%d days, %d posts, %d points, %.2f points per day\n",$timeflie
+s,$posts,$experience,$exprate;
my $nextlevel = 3000;
my $leveldays = ($nextlevel - $experience)/$exprate;
my $leveldate = $today->add(days => $leveldays);
printf "Level up in %d days, on %s\n", $leveldays, $leveldate->mdy();

[ww]

There are numerous homepages which utilize js to play pranks; there are numerous threads which detail ways to incorporate js into your profile here (or to link to an online but offsite server). I think search terms such as "nodelet" + "js" might be helpful.

And if that's a false trail, What are Nodelets? may offer some clues.