Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Parsing webpages

by tel2 (Pilgrim)
on Jan 28, 2013 at 03:14 UTC ( #1015621=perlquestion: print w/replies, xml ) Need Help??
tel2 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I'm wanting to parse feedback webpages using Perl, extracting some key fields, from an auction site which is a bit like eBay. Here's some sample input data:

As you can see from the right hand side of this webpage, this member has 4 pages of feedback, and I plan to iterate over each of them. For this, I intend to change the "&page=1" part of the URL from 1 to 4, and I can handle that part.

The fields I want to extract are:
- Type of feedback. I'll start by just taking the name of the gif file displayed, i.e. "happy_face1.gif", "neutral_face1.gif" or "sad_face1.gif".
- Name of member giving feedback (e.g. "david_parmenter").
- Date of feedback (e.g. "12 Nov 2012").
- Feedback comment (e.g. "good trade").
- Auction number (e.g. "523912464").

I'll be combining the above fields into records, which might initially look something like this:
sad_face.gif|david_parmenter|12 Nov 2012|good trade|523912464
And I'll get all the records on each feedback page for that member, to make a file.

What I want to know is:

1. What is a good way to parse the HTML pages? I know there are modules like HTML::Parser, HTML::PullParser, HTML::TreeBuilder, etc, but I haven't really used them, and don't want to waste time trying to work out which is best for this kind of task, so if you have experience with such, I'd appreciate your advice. (I know this could be done without such a module, too, and in the past that's how I've usually done that, but this webpage layout is quite complex so maybe a module will make life much easier.)

2. I don't need a full script, but I'd like to see the part which finds/extracts the above mentioned key fields I'm after, please. Ideally this should include extracting the list of page numbers between the ">>" & "<<" signs, which I could pretty easily do without a module, but I'm interested to see how to do it with one.

Thanks. tel2

Replies are listed 'Best First'.
Re: Parsing webpages
by CountZero (Bishop) on Jan 28, 2013 at 07:29 UTC
    "Trade Me" has a published API and it will be much easier to use this API rather than scrape the site.

    Actually, using the API is the only authorised way to automate access to the website:

    4.1.c You may not use a robot, spider, scraper or other unauthorised automated means to access the Website or information featured on it for any purpose.


    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics
      Thanks CountZero,

      Good points.  Didn't realise that.

Re: Parsing webpages
by Anonymous Monk on Jan 28, 2013 at 04:47 UTC
      Thank you Annonymous Monk.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1015621]
Approved by davido
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (6)
As of 2018-06-22 20:07 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (124 votes). Check out past polls.