Re: Problem in using Web::Scraper, coming from HTML::TreeBuilder::XPath

[kcott]

G'day sbasbasba,

Welcome to the monastery.

"Can't locate object method "new" via package "HTML::TreeBuilder::XPath" at /System/Library/Perl/Extras/5.10.0/Web/Scraper.pm line 115, <F1> line 1."

That pathname (/System/Library/Perl/...) indicates your OS is Mac OS X and you're using the system Perl (i.e. the version of Perl installed by Apple for its own use). You have Web::Scraper installed in /System/Library/Perl/Extras/5.10.0/: this means you've modified your system Perl; I don't know what other modifications you've made. It's generally not a good idea to alter the system Perl. See the responses to "Are there any major Perl issues with Mac OS X Lion?": I posted this question a couple of years ago when I first started using Perl on a Mac; I chose the perlbrew option (and have no problems after 2 years of use and multiple Perl upgrades). I'd recommend you look into perlbrew or an equivalent solution.

Your current problem is probably related to if, and where, you have HTML::TreeBuilder::XPath installed. You may also have other versions of Perl installed. Without more information, I can only provide troubleshooting tips:

• The shebang line at the start of your script (#!/usr/bin/perl) indicates that the system Perl (/usr/bin/perl) should be used to run the script. Use the "which perl" command to see if that's the default Perl: you may get something like /opt/local/bin/perl if you've installed MacPorts.
• Find out where your Perl module libraries are. "perl -V" will list these under @INC: for the default Perl; for a specific Perl, use a full pathname, e.g. "/opt/local/bin/perl -V".
• Determine how you're installing Perl modules. cpan is a fairly typical utility for this. Use "which cpan" to see the full path and compare with paths to perl.
• Search for .../HTML/TreeBuilder/XPath.pm on your system. Assuming it has been installed, it's probably in one of the paths listed under @INC:. If you can't find it, install it; if it's in an unexpected place, don't try to copy or move it, reinstall it.
• Your problem might be fixed by changing your shebang line to whatever your default Perl is. For all my scripts, I use "#!/usr/bin/env perl" which automatically uses the current default.

[Aside: I noticed you removed part of your original post and replaced it with new content. Please don't do this: it often invalidates comments already made; it can also be useful to subsequent readers to see what was considered and then discarded (i.e. others can learn from your mistakes). The correct way to deal with this is described in "How do I change/delete my post?".]

[sbasbasba]

Hi All,

Thanks for the great answers, and sorry for not having followed the right etiquette of this website. I am not a programmer, and I started using perl only two weeks ago.

So: I have tried to locate where perl is installed: /usr/bin/perl. However, I also see perl5.8.9 and 5.10.0. Furthermore, for some reasons cpan won't let me install modules, so I have basically been copy-pasting the source of the .pm module files I needed, and put them in the relative folders (i.e. HTML/TreeBuilder folder for XPath.pm). Most of these folders, I have no idea why, are located in /System/Library/Perl/Extras/5.10.0/, so that's where I put the files. Both XPath and Scraper are in subfolders of this folder. I know this is not how I should work, but I just couldn't figure out anything better when I started working and only now I realize that I probably made a huge mess.

What would you guys recommend to solve this problem? Where should the folders be located?

Thanks a lot for your precious help!

[Anonymous Monk]

What would you guys recommend to solve this problem? Where should the folders be located?

step 1) restore your system perl to the pristine condition that it was ; I can't help with exact steps, its macness :)

step 2a) maintain your own perl (easy) ( install CitrusPerl), this way you get a newer version of perl, and if apple-macness updates the system perl, you don't have to reinstall/recompile any modules, your perl remains untouched

step 2b) or maintain your own PERL5LIB with cpanm (like this cpanm --local-lib PERL5LIB export PERL_MB_OPT=--install_base /home/user/devstuff set PERL_MM_OPT=INSTALL_BASE=/home/user/devstuff)

get cpanm and install Web::Scraper into /home/username/myperllibs

curl -L http://cpanmin.us | perl - -v --local-lib /home/username/myperllibs App::cpanminus ExtUtils::MakeMaker Module::Build Web::Scraper

wget -O - http://cpanmin.us | perl - -v --local-lib /home/username/myperllibs App::cpanminus ExtUtils::MakeMaker Module::Build Web::Scraper

Then you can run perl -I/home/username/myperllibs/lib/perl5/CGI mypythonic.pl or

export PERL5LIB=/home/username/myperllibs/lib/perl5:$PERL5LIB
export PATH=/home/username/myperllibs/bin;$PATH

perlbrew or perlall are workable alternatives somewhat automating one or more of the above steps

But when apple-macness updates the system perl, you'll have to recompile/reinstall any modules with binary components(.so/.xs files) not provided by apple-macness (anything with .xs/.sp files in your myperllibs)

step 3) If you were using a "word processor" switch to a "notepad" equivalent, or a programmers editor ( scite, textpad, gvim, emacs, padre )

[Anonymous Monk]

grr, typo (rather obvious but still) -I/home/username/myperllibs/lib/perl5/CGI should be -I/home/username/myperllibs/lib/perl5

[sbasbasba]

The problem is I have no idea how to restore my Perl. I paid no attention at the beginning when installing modules, and I can't find anything online. I am sure both Web::Scraper and XPath are in the Perl folders under @INC{}; then why why why don't they work???