Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Batch remove URLs

by marto (Archbishop)
on Oct 27, 2017 at 11:31 UTC ( #1202139=note: print w/replies, xml ) Need Help??


in reply to Batch remove URLs

Link rot, what a pain. Here's something very quickly thrown together based upon your expanded criteria.

  • Create a new directory, copy your htm files in there.
  • Install the required modules, run the following from the command prompt: cpanm Mojolicious Path::Tiny.

  • Download the code below to the same location. Run the code.

This code reads the content of each htm file in a directory, parses it with Mojo::DOM, finds all links, checks each URL with Mojo::UserAgent , if it looks like it's dead it'll remove the parent HTML element. Saving the file after.

Example HTML:

<html> <head> <title>test</title> </head> <body> <ul> <li><a href="http://perlmonks.org">perlmonks</a></li> <li><a href="http://archive.org">archnive.org</a></li> <li><a href="http://sitedoesnotexist9999.net">fakesite</a></li> </ul> </body> </html>

Perl code:

#!/usr/bin/perl use strict; use warnings; use feature 'say'; use Path::Tiny; use Mojo::DOM; use Mojo::UserAgent; # get current directory my $dir = Path::Tiny->cwd; # for each html file for ( $dir->children( qr/\.htm$/ ) ){ # read the contents into a variable my $html = path( $_->basename )->slurp; # get the dom my $dom = Mojo::DOM->new( $html ); # find all links for( $dom->find('a')->each ){ # get target href my $url = $_->attr('href'); say "checking link $url"; # use Mojo::UserAgent to check if link is alive my $ua = Mojo::UserAgent->new; my $res; eval { $res = $ua->max_redirects(5)->head( $url )->result }; # if an error is thrown if ( $@ ){ warn "$url seems dead, removing parent"; $_->parent->remove; } # play nice sleep(10); } # save file path( $_->basename )->spew($dom->content); }

Example HTML after running program:

<html> <head> <title>test</title> </head> <body> <ul> <li><a href="http://perlmonks.org">perlmonks</a></li> <li><a href="http://archive.org">archnive.org</a></li> </ul> </body> </html>

Since I don't have an example of what you're actually using, and things like [404 Not Found] don't often make sense to keep around, I removed them, however simply using the replace method rather than remove on the parent does exactly what you want:

#!/usr/bin/perl use strict; use warnings; use feature 'say'; use Path::Tiny; use Mojo::DOM; use Mojo::UserAgent; # get current directory my $dir = Path::Tiny->cwd; # for each html file for ( $dir->children( qr/\.htm$/ ) ){ # read the contents into a variable my $html = path( $_->basename )->slurp; # get the dom my $dom = Mojo::DOM->new( $html ); # find all links for( $dom->find('a')->each ){ # get target href my $url = $_->attr('href'); say "checking link $url"; # use Mojo::UserAgent to check if link is alive my $ua = Mojo::UserAgent->new; my $res; eval { $res = $ua->max_redirects(5)->head( $url )->result }; # if an error is thrown if ( $@ ){ warn "$url seems dead, updating link"; $_->replace('[404 Not Found]'); } # play nice sleep(10); } # save file path( $_->basename )->spew($dom->content); }

Which outputs:

<html> <head> <title>test</title> </head> <body> <ul> <li><a href="http://perlmonks.org">perlmonks</a></li> <li><a href="http://archive.org">archnive.org</a></li> <li>[404 Not Found]</li> </ul> </body> </html>

There's a 10 second sleep in there, don't batter URLs. There is room for optimisation, for example if the same URL occurs more than once per file, a list of tested working URLs etc, but I'll leave that as an exercise for you.

Update: small addition.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1202139]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2019-05-26 16:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you enjoy 3D movies?



    Results (153 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!