Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^2: Help With Online Table Scraper

by Anonymous Monk
on Mar 03, 2011 at 09:40 UTC ( #891181=note: print w/replies, xml ) Need Help??

in reply to Re: Help With Online Table Scraper
in thread Help With Online Table Scraper

Sinistral is right, use a documented API whenever available, scraping is a fragile PITA :)
#!/usr/bin/perl -- use strict; use warnings; use Data::Dumper; use Web::Scraper; use File::Slurp qw' read_file '; use URI; Main( @ARGV ); exit( 0 ); sub Main { my $uri = URI->new(' +stics'); if(@_){ my $html_content = read_file( @_ ); ScrapePrint(\$html_content, $uri ); } else { ScrapePrint( $uri ); } } sub ScrapePrint { my $scraper = scraper { process q~//table[@id='yfncsumtab']/tr/td/table[8]//tr//tr~, 'Balance Sheet[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process #~ q~//table[@id='yfncsumtab']/tr/td/table[9]//tr//tr~, q~//table[@id='yfncsumtab']//table[9]//table//tr~, 'Cash Flow Statement[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr[2]/td[3]/table[4]//tr//tr~, 'Dividends & Splits[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr/td/table[4]//tr//tr~, 'Fiscal Year[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr/td/table[7]//tr//tr~, 'Income Statement[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '//tr/td[2]', 'value.', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr/td/table[6]//tr//tr~, 'Management Effectiveness[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr/td/table[5]//tr//tr~, 'Profitability[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr/td[3]/table[3]//tr//tr~, 'Share Statistics[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr[2]/td[3]/table[2]/tr//tr~, 'Stock Price History[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, process q~//table[@id='yfncsumtab']/tr[2]/td/table[2]/tr/td/table/ +tr~, 'Valuation Measures[]' => scraper { process '/tr/td[1]', 'key', 'TEXT'; process '/tr/td[2]', 'value', 'TEXT'; }, }; my $res = $scraper->scrape( @_ ); print Data::Dumper->new([ $res ])->Sortkeys(1)->Dump; } __END__ $VAR1 = { ... 'Cash Flow Statement' => [ { 'key' => 'Cash Flow Statement' }, { 'key' => 'Operating Cash Flow (ttm):', 'value' => '6.30M' }, { 'key' => 'Levered Free Cash Flow (ttm):', 'value' => '3.66M' } ], ...
Come to think of it, Web::Scraper might be also be a bit of a PITA, but I've only studied the trivial examples, not the others

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://891181]
[choroba]: I already gave a similar talk to my friends in a pub and at an internal conference at work and people liked it, so maybe...
[choroba]: LanX: That's the heritage, I can't do anything else
[RonW]: Sounds like some system my employer has "It does exactly what we need it to do and can't afford to risk anything we can't prove is 100% compatible"
[marto]: choroba sounds interesting
[RonW]: james28909 Why not write a Perl program to do the task?
[choroba]: RonW Yes, but then, one day, they needed to switch from FTP to SFTP, and... but I can't give the whole talk away here :)
[RonW]: Tunnel FTP through stunnel?

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (10)
As of 2017-05-22 21:39 GMT
Find Nodes?
    Voting Booth?