Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Capturing web pages and making them static

by Biff (Novice)
on Aug 04, 2004 at 21:03 UTC ( #380120=perlquestion: print w/replies, xml ) Need Help??
Biff has asked for the wisdom of the Perl Monks concerning the following question:

Before I ask my question here's a little background in case I'm just being stupid in my attempted implementation. We have reports that we need to make available to manglers :) but the reports are big and slow, etc. The poor manglers get impatient and un-happy.

My mission is to generate these reports at night and save static copies of them for the mangler's viewing pleasure.

What I'm trying to emulate is the capability that most browser's have of saving a web page as a complete static entity. When you do this in a browser you get the page saved as HTML and a folder full of goodies and other artifacts that enable you to view the page statically just like you would have off of the server.

Will a simple LWP get do this? I kinda doubt it and was hoping to avoid blind alleys.



  • Comment on Capturing web pages and making them static

Replies are listed 'Best First'.
Re: Capturing web pages and making them static
by perrin (Chancellor) on Aug 04, 2004 at 21:24 UTC
    You can use LWP, probably with the lwp-mirror tool, or you can use wget or curl.
Re: Capturing web pages and making them static
by Aristotle (Chancellor) on Aug 04, 2004 at 22:20 UTC

    This is not exactly trivial. In addition to having to fetch the pages, you need to parse them, find all the links to additional resources, download these resources, and change the links to point to the local copies.

    You probably don't want to write this yourself. Of the tools available on most any Unix box, wget is capable of doing this for you. If you want something written in Perl, try w3mir.

    Makeshifts last the longest.

Re: Capturing web pages and making them static
by MidLifeXis (Monsignor) on Aug 04, 2004 at 21:05 UTC

    If you use Apache, look at mod_proxy. IIRC, there is an example in the documentation on how to implement exactly your problem.


Re: Capturing web pages and making them static
by Fletch (Chancellor) on Aug 04, 2004 at 23:33 UTC

    Erm, if you're generating the pages yourself just rewrite things to output the HTML to a file. Or even if you don't want to change the CGIs just write a wrapper script which sets the appropriate environment variables ($ENV{QUERY_STRING}=q{foo=bar&zagnork=wubble};) and then runs the CGI and redirects the output into a file. Then you run your wrapper from cron and save the results somewhere under your document root.

Re: Capturing web pages and making them static
by GaijinPunch (Pilgrim) on Aug 04, 2004 at 23:41 UTC
    I am actually working on something similar to this myself (except I parse the info and email what I want to msyelf.) If you're wanting to simply grab a webpage from some place on the net and save it, LWP will work perfectly.
    use LWP::Simple; my $url = ""; my $webpage = get( $url );
    Then just write $webpage somewhere, and you've got a "saved" html file. I think this is what you're asking for. EDIT: I don't think LWP::Simple will save header information though, which may or may not be an issue for you.
Re: Capturing web pages and making them static
by johnnywang (Priest) on Aug 05, 2004 at 00:03 UTC
    A few other CPAN modules might be useful: WWW::Mechanize and WWW:WebRobot. I've only used these for automated tests. I guess in your case, you still need to change the link in the original page (say images) to local copies. w3mir can be used for downloading a whole subtree, most likely can't do exactly what you want since your page can (theoretically) include an image from somewhere else.
      Interesting -- do any of the above modules to the image name changing for you? A favorite game developer of mine is reworking their entire website, and was thinking of backing it up before it got too restructured... they leave a lot of good information out sometimes.
Re: Capturing web pages and making them static
by Wassercrats on Aug 05, 2004 at 01:14 UTC
    I read part of the synopsis for w3mir at I wish people who don't write English well would have someone edit the documentation for them.

    This part makes setup sound complicated:

    For authentication and passwords, multiple site retrievals and such you will have to resort to a "CONFIGURATION-FILE". If browsing from a filesystem references ending in '/' needs to be rewritten to end in '/index.html', and in any case, if there are URLs that are redirected will need to be changed to make the mirror browseable, see the documentation of Fixup in the "CONFIGURATION-FILE" secton.

    w3mirs default behavior is to do as little as possible and to be as nice as possible to the server(s) it is getting documents from. You will need to read through the options list to make w3mir do more complex, and, useful things.

    People are quick to say not to reinvent the wheel, but they'll still tell you to deal with modules when there are complete solutions available. Look at

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://380120]
Approved by Plankton
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2018-02-17 23:39 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (250 votes). Check out past polls.