in reply to
With some prodding from shmem, I was able to make progress, culminating in the code post above for "Update 2".
However, I still can't match the text I want, becuase the page fetched with WWW::Mech differs from the text I get when I open the url in firefox and do view source.
At any rate, to simply reiterate the test case for my problem (basically the same as Update 2 above, but shorter and simpler):
my $price = '249,9';
print "price: $price\n";
my $url = 'http://www.baur.de/is-bin/INTERSHOP.enfinity/WFS/Baur-BaurD
my $mech = WWW::Mechanize->new();
$mech->agent('Firefox/1.0 (Windows; U; Win98; en-US; Localization; rv:
+1.4) Gecko/20030624 Netscape/7.1 (ax)');
$mech->get( $url );
my $html = $mech->content;
print "html from $url doesn't match $price\n" unless $html =~ /$price/
print "but paste into browser and view source, and it does\n";
print "final url after firefox redirect (but not www::mech redirect) i
+s something like " . 'http://www.baur.de/is-bin/INTERSHOP.enfinity/WF
+lse' . "\n";
# uncomment this to print html, which is totally different from what y
+ou get from firefox, show source.
# print "html: $html";
Can anyone prod me further in the right direction?
PS: I'm still wondering if I would get better results with Win32::IE::Mechanize or WWW::Selenium or Mozilla::Mechanize.
UPDATE: better user agent string after perrin comment.