|The stupid question is the question not asked|
web scraper testingby alienhuman (Pilgrim)
|on Apr 11, 2006 at 20:52 UTC||Need Help??|
alienhuman has asked for the
wisdom of the Perl Monks concerning the following question:
I've got a web scraping script. It works fine, except it's a PITA to test the program logic, because the conditions under which it scrapes an external web site only happen for about an hour a day (the information being scraped is time sensitive).
In order for me to test the script's logic outside of that one hour a day, I have to fake:
I currently accomplish this by setting a "TEST" flag in my code, and at certain junctures testing for it and running different code if I'm testing. Then there's also some bits of code that I just comment/uncomment during tests. I'd like to rewrite the package that contains my obj/methods so that when the object is created as a "test", the usual methods will not scrape the external site, query the DB, etc normally. Instead they'll execute under testing conditions, so that I can test program logic during other times of the day.
Any thoughts on how, generally, to think about writing code to handle this kind of thing?
Thanks in advance,