Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
Welcome to the Monastery
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
Since nobody else has commented, I will add mine. WONDERFUL!

I've got a few ideas for this, and it actually is very close to the spider I've been working on for Plucker in my spare time. Bear with me while I brain-dump these out:

  1. LWP::Parallel::UserAgent to fetch content asynchronously.
  2. HTTP::Cookies for storing client-side cookies into the "jar"
  3. Link rewrite rules, so gathering/spidering the content locally after fetching can be done with relative/absolute links, without breaking anything. (URI::URL can help here)
  4. Ability to forge Referer and UserAgent strings through the use of --referer and --useragent. Trivial to add.
  5. More verbose progress reporting (use LWP::Debug qw(+);)
  6. Options for staying on the same host, same domain, or staying below a certain fragment of the URI. Something like:
    # url http://www.domain.com/foo/bar/blort/quux --staybelow http://www.domain.com/foo/bar/ --stayonhost www.domain.com --stayondomain domain.com
  7. Ability to update the "cache" on multiple runs. Compare the remote file with the local file, and fetch if newer.

Expect some patches from me to come flying in within the next few weeks on this one. Great work!


In reply to Re: goose.pl by hacker
in thread goose.pl by elusion

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (7)
    As of 2014-04-23 23:52 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (557 votes), past polls