Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: Perl :: Mechanize - running a single while loop

by Anonymous Monk
on May 19, 2011 at 18:58 UTC ( #905778=note: print w/ replies, xml ) Need Help??


in reply to Perl :: Mechanize - running a single while loop

Something like

GetSome( starting url ); sub GetSome { my $mech ... my @pages = ... while(@pages) { my $page = shift @pages; $mech->get( $page ); push @pages, GetMorePages( $mech ); SomethingImportant( $mech ); SomethingXPATH( $mech ); } }
combine with Re: Help With Online Table Scraper.

All the interesting stuff happens in GetMorePages, SomethingImportant, and SomethingXPATH

Can you give me a hint for the beginning - the processing of the entry pages - doing this in Perl:: Mechanize

Before writing any code, make a nice flowchart, maybe with Text::Flowchart, of how you think your program should work.

Once you have an overall idea of how your program should work, once you have the skeleton (like I outlined above), then think about writing some code (like the skeleton).

Next part is to write a small program in the spirit of Re^5: WWW::Mechanize clicking checkboxes or Re^2: Help With Online Table Scraper (or Re^2: Printing just the file name for all the cases), to solve one part of your diagram, like GetMorePages.

Then incorporate GetMorePages into your main program, and then repeat these steps for some other part, like SomethingXPATH

And repeat until you're finished


Comment on Re: Perl :: Mechanize - running a single while loop
Download Code
many thanks!! Great points
by Perlbeginner1 (Scribe) on May 19, 2011 at 21:59 UTC
    Hello dear Friend,

    many thanks for the good ideas and hints! You raise valid points. Many thanks i will try to figure out all the necessary steps. Your posting helps alot!

    greetings

    pb1

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://905778]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2014-08-23 18:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (178 votes), past polls