Re: How do I Extract Data From Amazon.com's Website?

[matsi]

Respected Monks suggested several modules aimed at handling Amazon site.
But if you want implement similar stuff yourself or probably you need to solve similar problem for another website, then you should look around LWP modules.

[Joost]

If you're dealing with a web interface (i.e. HTML with forms, links, etc instead of "pure" HTTP) you're probably better off using WWW::Mechanize - it's an LWP subclass with lots of specialized methods to search and "click" through web pages/forms.

There are even a few "clones" of WWW::Mechanize that use popular browsers at the back-end so you can deal with javascript and other client-side objects not normally supported by WWW::Mechanize.

[erroneousBollock]

Recently, some folks figured out a way to use Mozilla in a mechanize driver without the need for a (visible) X-server.

Sounds promising.

-David