Best way to recursively grab a website

[ghenry]

Dear all,

I have a two part program that sits in the hooks folder of a subversion repo. After a commit, the first part needs to checkout, or download the latest version either with the svn command or via a standard subversion apache browse. The second part uploads the newly grabbed files to aberdeen.pm.org via perldav.

The second part is done.

The first part, I create a temporary folder with File::Temp and then move on. Should I use LWP for recursive downloads? I don't really want to have a load of system commands etc.

What module/feature have I blindly missed?

Thanks,

Gavin.

[gjb]

If you don't mind one system call, you could go with wget, an excellent tool to download an entire website. Command line option allow to restrict downloads to a single site, a certain depth and what not. All in all, a very valuable tool. It can be found at http://www.gnu.org/software/wget/wget.html.

Did I mention it's free software (a GNU project to be precise)?

Hope this helps, -gjb-

[ghenry]

I think that will be the easiest method.

Thanks.

[tlm]

WWW::Mechanize is my tool of choice for that sort of thing.

[ghenry]

Of course. I remember that now. I have read loads of articles about it, but it slipped my mind.

Cheers!

I'll post my code later.

[webchalkboard]

maybe i'm just not as techy as you guys, but that first bit was double dutch to me :)

you could try using wget, that's a program i've used in the past for mirroring websites, otherwise there is a mirror routine inthe LWP::Simple module on cpan which might do it.

http://search.cpan.org/~gaas/libwww-perl-5.803/lib/LWP/Simple.pm

mirror($url, $file) 
Get and store a document identified by a URL, using If-modified-since,
+ and checking the Content-Length. Returns the HTTP response code.

Hope that's some help.

[ghenry]

Yeah, sorry. I added the background of the problem, just so people who are familiar with Subversion, understand where I am coming from.

I have looked at LWP::Simple, but the mirror function looks like it is per-url, not per website?

I could use wget, but I want to do it all in perl, in case wget is not available.

Thanks.

[inman]

LWP ships with a number of example applications / utilities such as GET, POST etc. One of these utilities is lwp-mirror.

From the documentation:


This program can be used to mirror a document from a WWW server. The document is only transfered if the remote copy is newer than the local copy. If the local copy is newer nothing happens.