Category: PerlMonks Related Scripts
Author/Contact Info /msg jeffa
Description: Is your Personal Nodelet getting too big? Mine sure is was. So, I wrote this little bot to parse my Personal Nodelet and output the nodes in the form of: id [delimiter] name. Once the data is in an easy (and fast) form to parse, you can use another script to edit it, munge it some more - or you can modify this script to reflect your needs.

Also, check out Personal Nodelet Restorer.

Comments and improvements always welcome. :)

thanks to crazyinsomniac PodMaster for pointing out my lack of Win32 support in the synopsis

UPDATE: - after gleaming good security info from jcwren, i changed this script to accept an interactive password similiar to how mysql asks for passwords: you can specify the password on the command line with the -p option (dangerous for mulit-user boxes), or you can just specify -p by itself and be prompted for the password. I use Term::ReadKey to keep the password 'invisible'. Because of the new nature, be sure and specify -p last, after -u and -s if used.

UPDATE (Mar 24,2002): - somebody moved 'lastnode' before 'node' so regex has been updated

#!/usr/bin/perl -w
use strict;

use LWP;
use Term::ReadKey;
use Getopt::Std;
use HTML::TokeParser;
use HTTP::Request::Common;

$| = 1;

use vars qw(%opts);
my ($user,$passwd,$help) = parse_args(\%opts);
USAGE() and exit unless $user and $passwd and not $help;

use constant URL => '';
my $SEP = $opts{s} || "\t";

my $ua = LWP::UserAgent->new;
$ua->agent('personal_nodlet_extractor/1.0 (' . $ua->agent .')');

# log in and access your User Setting page in raw format
my $request = POST(URL,
   Content => [
      op          => 'login',
      user        => $user,
      passwd      => $passwd,
      node_id     => 1072,
      displaytype => 'raw',

my $response = $ua->request($request);
die $response->message unless $response->is_success;

# pass the HTML content to TokeParser
my $content = $response->content;
my $parser  = HTML::TokeParser->new(\$content);

# 'fast forward' until we find "<b>Personal Nodelet</b>"
while ($parser->get_tag('b')) {
   last if 'Personal Nodelet' eq $parser->get_text;

# these are the [links|droids] we are looking for
while (my $tag = $parser->get_tag('a')) {
   if ($tag->[1]->{href} =~ /[^t]node_id=(\d+)/) {
      print $1 . $SEP . $parser->get_text . "\n";

sub parse_args {
   my %opt = %{+shift};
   if (exists $opt{'p'} and not defined $opt{'p'} and defined $opt{'u'
+}) {
      print "Enter password: ";
      ReadMode 'noecho';
      chomp($opt{'p'} = ReadLine 0);
      ReadMode 'normal';
   return @opt{qw(u p h)};

sub USAGE { print "USAGE: $0 -u user [-s separator] -p password\n" }


=head1 NAME - LWP script


This is a simple script that uses LWP and HTML::TokeParser to
access your User Settings page and extracts your Personal
Nodelet links.


  for *nix:
    ./ -u uname -s : -p 
  for win32:
    perl -u uname -s : -p

Invokes the script for the specified username and use a 
colon as the record separator. The script will prompt
you for your password if you do not specify it. The
contents would look something like this:

 24270:Permutations and combinations
 25730:Life in the land of OOP, and I'm confused.
 17890:Shift, Pop, Unshift and Push with Impunity!
 34786:Why I like functional programming

The default record seperator is a tab. I recommend you use it
since just about any character is fair game for a node title.

Replies are listed 'Best First'.
Re: Personal Nodelet Extractor
by mirod (Canon) on Aug 03, 2001 at 18:58 UTC

    The default record seperator is a tab. I recommend you use it since just about any character is fair game for a node title.

    Maybe you could add an other option, -r for example that would replace $SEP:

    # option definition becomes(sorry to break the push ;--( getopts('p:u:s:r:h',\%opts); # replacement string initialization my $REP= $opts{r} || " "; # loop that processes the links while (my $tag = $parser->get_tag('a')) { if ($tag->[1]->{href} =~ /node_id=(\d+)/) { my $text= $parser->get_text; $text=~ {$SEP}{$REP}go; print $1 . $SEP . $text . "\n"; } }