Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re: Scraping AJAX?

by Anonymous Monk
on Jan 26, 2012 at 16:57 UTC ( #950165=note: print w/replies, xml ) Need Help??

in reply to Scraping AJAX?

can't figure out how to extract data
Learn XPath!
use WWW::Mechanize::Firefox qw(); use HTML::TreeBuilder::LibXML qw(); my $mech = WWW::Mechanize::Firefox->new; $mech->get(' +d/demo/datagrid-simple.html'); my $tree = HTML::TreeBuilder::LibXML->new; $tree->parse($mech->content); $tree->eof; my $name = $tree->findvalue('/html/body/div/div[2]/div/div/div/div/div +[2]/table/tbody/tr/td[2]');

Replies are listed 'Best First'.
Re^2: Scraping AJAX?
by Dave_COS (Initiate) on Jan 26, 2012 at 17:14 UTC

    I do know XPath.

    More details: I'm trying to scrape a Blackberry Administration Service webpage, which is behind a firewall; otherwise I would have given that link.

    I'm trying to get the element


    Which Chrome "Inspect element" sees, and the Selenium IDE sets every time -- but when I try to run against WWW:Mechanize it shows NO content, and the Selenium packages state that element does not exist.

    Forgive me if the solution is obvious, but I *have* read the other posts -- and tried the code -- and have had no success.

      Thats because you need to get() the JSON data which contains the actual table you want to scrape. The content you are getting is only the page that contains the JS functions and there should also be the ajax link in there you can use to get the actual data.

      Once you get that link, get() it then parse the data from there.

        Forgive me again if this is answered elsewhere --

        Is there a way to do this on a headless system? firefox insists on a display....

        System: Ubuntu server 10.04

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://950165]
[Discipulus]: and in the wanted sub you just calculate the duretion and add it the total, stop
[shmem]: sub wanted { push @files, $File::Find::name if $File::Find::name =~/\.mp3$/ }
[Lady_Aleena]: shmem, that is understandable! The two examples in File::Find don't make sense to me on a quick glance.
[marioroy]: LA if the find worked from Unix command line, or does it not. Likely a quoting issue inside Perl qx.
[Lady_Aleena]: marioroy, the find worked fine at the command line.
[marioroy]: LA, yeah. than there's no reason why it cannot work inside qx. But chatting is hard in PM. I cannot see the code now.
[shmem]: Lady_Aleena: sometimes a quick glance isn't enough.

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (9)
As of 2017-04-23 20:54 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (432 votes). Check out past polls.