Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
We don't bite newbies here... much
 
PerlMonks  

Struggling with a data feed/data scraping perl program

by jdlev (Scribe)
on Nov 16, 2013 at 02:05 UTC ( #1062838=perlquestion: print w/ replies, xml ) Need Help??
jdlev has asked for the wisdom of the Perl Monks concerning the following question:

Hey guys,

First of all, thanks to everyone who has helped me in the past! This site rocks the casba!! : )

I'm trying to figure out how to scrap data from yahoo's fantasy football site. It is very complex when looking at the page's source code.

Really, all I want to do, is be able to enter a week, and get the projections for that player. Here is the page I am trying to pull the data from. Once I have the data, I can insert into one of my MySQL databases. Can someone point me in the right direction on how to scrap the players first name, last name, and projected points from this web page:

http://football.fantasysports.yahoo.com/f1/628117/players?&sort=PR&sdir=1&status=ALL&pos=O&stat1=S_PW_11&jsenabled=1&jsenabled=1

Thanks a ton in advance!!! :)

I love it when a program comes together - jdhannibal

Comment on Struggling with a data feed/data scraping perl program
Re: Struggling with a data feed/data scraping perl program
by 2teez (Priest) on Nov 16, 2013 at 06:27 UTC

    Hi jdlev,
    ..Can someone point me in the right direction on how to scrap the players first name, last name, and projected points from this web page..
    You may consider using these two modules LWP::UserAgent and HTML::TreeBuilder to get and parse through the web page for the information you want.
    I couldn't get the page you gave, but use one to illustrate, like thus:

    use warnings; use strict; use LWP::UserAgent; use HTML::TreeBuilder 5 -weak; my @players_name; my $url = 'http://facebook-football.fantasysports.yahoo.com/f1/745190/players?st +atus=A&pos=O&cut_type=9&stat1=S_S_2013&myteam=0&sort=PR&sdir=1'; my $ua = LWP::UserAgent->new; my $broswer = $ua->get($url); if ( $broswer->is_success ) { my $tree = HTML::TreeBuilder->new; $tree->parse_content( $broswer->decoded_content ); @players_name = $tree->look_down( 'class' => qr/Nowrap name/ ); } else { die $broswer->status_line; } print $_->as_text, $/ for @players_name;
    The code above will print all the firstname and lastname of the players.
    There are other modules in CPAN or METACPAN that can also do so. So, check in and get started, now you have the right direction you wanted :)

    If you tell me, I'll forget.
    If you show me, I'll remember.
    if you involve me, I'll understand.
    --- Author unknown to me
Re: Struggling with a data feed/data scraping perl program
by afoken (Parson) on Nov 16, 2013 at 09:10 UTC
    Can someone point me in the right direction on how to scrap the players first name, last name, and projected points from this web page

    Search the website for a public API. Ripping data from a heap of HTML junk is the worst way of aquiring data.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re: Struggling with a data feed/data scraping perl program
by karlgoethebier (Deacon) on Nov 16, 2013 at 12:53 UTC

    Perhaps you should take a look at this?

    Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1062838]
Approved by 2teez
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (16)
As of 2014-04-18 20:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (471 votes), past polls