Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: Foreach loop help. Start from specific line

by sundialsvc4 (Abbot)
on Oct 22, 2012 at 13:42 UTC ( #1000344=note: print w/ replies, xml ) Need Help??


in reply to Foreach loop help. Start from specific line

Every script needs to be restartable.   If it were me, I would have put the million names into, say, an SQLite database file and include a “processed?” column which gets updated each time.   I might put summary results of some kind in there also.   Such an approach would not only handle the restart issue, but would also be good for handling the re-run issue.   You will from time to time need to re-query those websites and the script will need to be able to know how to do that, too.


Comment on Re: Foreach loop help. Start from specific line
Re^2: Foreach loop help. Start from specific line
by Doozer (Beadle) on Oct 22, 2012 at 14:18 UTC
    That is a good point. I am fairly new to perl and this script is the most complex one I have written so far (It branches off sending ssh commands to other Virtual Machines to start counterpart scripts which then loop back so it is all closed loop).

    You are correct about re-querying the websites. We log every hostname that does not respond with '200 OK' to a separate file which will then be run through the test process again.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000344]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (10)
As of 2014-10-23 08:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (125 votes), past polls