Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Re^2: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)

by brengo (Acolyte)
on Nov 26, 2010 at 00:27 UTC ( #873745=note: print w/replies, xml ) Need Help??

in reply to Re: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)
in thread how to quickly parse 50000 html documents?

Wow. Just wow. Thank you for these lines and great ideas ("discard first half of matches" and use regex)!

Just a small thing: the regex gives me the whole line instead of just the values back when running it (what did I miss?):

$ perl -nle"m[<font size=1>([^<]+)</font></td></tr>] and print $1" jun +k.html <tr bgcolor=#DFDFDF><td><font size=1>drill diameter:</font></td> + <td +><font size=1>936</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>drill depth:</font></td> + <td><font s +ize=1>20</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>drill speed:</font></td> + <td><font size=1>4</font></ +td></tr> <tr bgcolor=#CCCCCC><td><font size=1>drill material:</font></td> + <td +><font size=1>506</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>height:</font></td> + <td><font s +ize=1>502</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>width:</font></td> + <td><font size=1>6</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>angle:</font></td> + <td><font size=1>2.76</font +></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>cooling liquid:</font></td> + <td><font s +ize=1>14</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>manufactured in:</font></td> + <td +><font size=1>27</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>lane code:</font></td> + <td><font size=1>76 +</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>quality test 1:</font></td> + <td +><font size=1>581 (11.4%)</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>quality procedure:</font></td> + <td +><font size=1>19,021</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>quality test 2:</font></td> + <td><font s +ize=1>843 (90.1%)</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>package worth:</font></td> + <td><font s +ize=1>$257,524</font></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>single unit worth:</font></td> + <td +><font size=1>$90,945</font></td></tr> <tr bgcolor=#CCCCCC><td><font size=1>colour:</font></td> + <td><font size=1>48</font>< +/td></tr> <tr bgcolor=#DFDFDF><td><font size=1>coating:</font></td> + <td><font size=1>2,602</fon +t></td></tr> <tr bgcolor=#DFDFDF><td><font size=1>sold this month:</font></td> + <td><font size=1>118</font> +</td></tr>
  • Comment on Re^2: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)
  • Download Code

Replies are listed 'Best First'.
Re^3: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)
by BrowserUk (Pope) on Nov 26, 2010 at 00:54 UTC

    Going by your prompt, you are running on some kind of *nix system; in which case you should swap the "s for 's.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://873745]
[beech]: checking chatterbox sidebar
[beech]: and again a check on chatterbox sidebar, what is expandability

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (6)
As of 2017-03-27 22:50 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (324 votes). Check out past polls.