http://www.perlmonks.org?node_id=1002919

ati has asked for the wisdom of the Perl Monks concerning the following question:

This node falls below the community's threshold of quality. You may see it by logging in.

Replies are listed 'Best First'.
Re: fix the problem of the web crawler
by frozenwithjoy (Priest) on Nov 08, 2012 at 16:39 UTC
    Three hints (and a suggestion):
    1. The URLs that the script is generating are correct.
    2. The regex doesn't seem to be matching anything because the style code has changed on the website.
    3. There are lots of other potential problems with your script that can be found with use strict; use warnings;
    4. Considering you posted nearly the same wall of script a year ago, it might be worth paying someone to clean it up and make it work properly.

    Edit: Is this how the output (conf.txt.) is supposed to look? (I accept all PayPal alternatives... Just kidding... Sort of... But seriously, if this is the expected output and you follow my hints, you'll figure it out.)

    1=James F. Blakesley=Frederick H. Wolf 
1=James F. Blakesley=Keith S. Murray 
1=James F. Blakesley=Dagmar Murray 
2=James F. Blinn=Turner Whitted 
2=James F. Blinn=Pat Hanrahan 
2=James F. Blinn=Tomas Porter 
2=James F. Blinn=Flip Phillips 
2=James F. Blinn=Martin E. Newell 
2=James F. Blinn=Jeffrey M. Lane 
2=James F. Blinn=Nick England 
2=James F. Blinn=Loren C. Carpenter 
2=James F. Blinn=Alvy Ray Smith 
2=James F. Blinn=Donna J. Cox 
2=James F. Blinn=Helga M. Leonardt Hendriks 
2=James F. Blinn=Charles T. Loop 
2=James F. Blinn=Rob Pike 
2=James F. Blinn=Richard Ellison 
3=James F. Blowey=John W. Barrett 
3=James F. Blowey=Stephen Langdon 
3=James F. Blowey=John R. King 
4=James F. Bowring=Mary Jean Harrold 
4=James F. Bowring=James M. Rehg 
4=James F. Bowring=Alessandro Orso 
4=James F. Bowring=James A. Jones

    [download]
[reply]
[d/l]
[select]

      It is the exact output I've had before.

[reply]
        Here are a couple more (very specific) hints:
        1. Uncomment out the print page line so you can see the content you are scraping (or just go to the appropriate URL and view source).
        2. Change this part of the regex since it is apparently out-of-date: <td\sclass="coauthor"\salign="right"\sbgcolor="[^"]+">

        Also, I don't mean to be a jerk, but it is really better for you if you work through this yourself. Instead of sending me messages, you should show what you are trying here and people will be more willing to help when they've seen that you are indeed making a noble effort. Like the ancient saying goes: "Monks help those that help themselves!"

[reply]
[d/l]
[select]
Re: fix the problem of the web crawler
by bitingduck (Chaplain) on Nov 08, 2012 at 16:29 UTC

    In my limited experience with screenscrapers, that failure mode is usually caused by someone at the other end changing the formatting. You're looking for the stuff you want with a regex, rather than an html parser, so you can easily be a victim of very minor changes in the html. Your best bet is probably about 10 minutes of looking at the page source and then revising the regex accordingly. Switching to using HTML::TreeBuilder and taking advantage of predictable page structure and tag attributes might make your script a little more robust (or it might not, depending on who is messing with it at the other end...). I have a scraper that's been running reliably for several years now through a number of changes in the target page's display format since I switched to treebuilder.

[reply]
Re: fix the problem of the web crawler
by zwon (Abbot) on Nov 08, 2012 at 15:28 UTC

    If you think this is a place there you're posting your non-working scripts and they get fixed, it is not. There's a lot of freelance websites where you can find a programmer to fix your problem.

[reply]

Back to Seekers of Perl Wisdom