Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re: best approach to parse vichan-style imageboard html (dom/css xpath)

by beech (Parson)
on Mar 18, 2019 at 00:02 UTC ( #1231380=note: print w/replies, xml ) Need Help??


in reply to best approach to parse vichan-style imageboard

Hi,

You do not want to use anything with "parser" in the name

#!/usr/bin/perl -- use strict; use warnings; use Mojo::DOM; my $html = q{ <div class="post reply body-not-empty" id="reply_8735435"> (cut out for visibility) <p class="body-line ltr ">The first 3 lines were 15% bait power, but t +hen it fell to mere 5% and the last lines are literally 0%, try again + in a few days.</p> </div> (cut out for visibility) <div class="post reply body-not-empty" id="reply_8735439"> (cut out for visibility) <div class="body" > <p class="body-line ltr "> <a onclick="highlightReply('8735417', event);" href="/b/res/8735417.ht +ml#8735417">&gt;&gt;8735417</a> </p> <p class="body-line ltr quote">&gt;Reddit is a great place for discour +se and there are many active subreddits where field professionals reg +ularly answer questions on issues of health, science, engineering, et +c</p> <p class="body-line ltr ">Yeah, as far as content goes, Reddit kicks 8 +chan's ass. They have some great boards for serious academic discussi +on.</p> <p class="body-line empty"> }; my $dom = Mojo::DOM->new( $html ); for my $e ( $dom->find( 'div.reply' )->each ){ print $e->{id},"\n", $e->text, "\n\n"; } __END__ reply_8735435 (cut out for visibility) reply_8735439 (cut out for visibility)
  • Comment on Re: best approach to parse vichan-style imageboard html (dom/css xpath)
  • Download Code

Replies are listed 'Best First'.
Re^2: best approach to parse vichan-style imageboard html (dom/css xpath)
by lis128 (Novice) on Mar 18, 2019 at 07:14 UTC

    I must admit that your approach is way clearer and understandable than mine.
    Once done with current problem i'll try to read plain HTML using given method :)
    One thing i don't get is statement "avoid parsers": why so?
    I was convinced that parsers were ment to extract data from more complext structures

    On the other hand i've managed to access chans using JSON interface (following https://github.com/vichan-devel/vichan-API/), but my question remains valid for HTML-only imageboards

Re^2: best approach to parse vichan-style imageboard html (dom/css xpath)
by karlgoethebier (Monsignor) on Mar 18, 2019 at 16:20 UTC
    "You do not want to use anything with "parser" in the name"

    At least the magic word is in the description of Mojo::DOM:

    "Mojo::DOM is a minimalistic and relaxed HTML/XML DOM parser with CSS selector support. It will even try to interpret broken HTML and XML..."

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1231380]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (3)
As of 2020-02-21 17:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What numbers are you going to focus on primarily in 2020?










    Results (96 votes). Check out past polls.

    Notices?