Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: WWW::Scripter performance and warnings

by PerlSufi (Friar)
on Dec 12, 2013 at 15:57 UTC ( [id://1066868]=note: print w/replies, xml ) Need Help??


in reply to WWW::Scripter performance and warnings

I'm not totally clear on why you are using WWW::Scripter as opposed to WWW::Mechanize- except that WWW::Mechanize doesn't work with javascript very well. However, I was able to connect to that page just by doing the following:
use WWW::Mechanize; use strict; use warnings; my $mech = WWW::Mechanize->new(); $mech->get('http://www.immoweb.be/FR/Rent.Estate.cfm?IdBien=2805206&xp +age=1'); $mech->dump_text;
Additionally, if you want to crawl the site AND use some of the javascript, you can either:
A) use WWW::Mechanize::Firefox
B) inspect the various html elements of the page with Firefox's firebug extension and use the $mech->get() similar to what I did above
UPDATE:
C) Or Go with Laurent_R's response ;)

Replies are listed 'Best First'.
Re^2: WWW::Scripter performance and warnings
by bretelle (Initiate) on Dec 16, 2013 at 10:57 UTC
    Hi PerlSufi,

    one element I need in the web page is printed there by a Javascript script. So I need WWW::Scripter (or something else) to execute this script.

    I guess Scripter is slow because there is a lot of Javascript in this page.

    When I inspect this particular element with Firebug I can see the name of the script. The question now is whether I can use that information in my Perl script so that WWW::Scripter executes only that one script. I'll first have to try to understand a bit more about Javascript and WWW::Scripter.

    Thanks a lot for your answers

    UPDATE I also tried solution A, WWW::Mechanize::Firefox, which does the work OK but is not faster, more than one minute to perform get().

      I guess Scripter is slow because there is a lot of Javascript in this page.

      Or infinite loop, memory leaks .... even the browsers (firefox/chrome...) do very little to protect from this

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1066868]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2024-03-28 08:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found