Re^9: Need help with WWW::Mechanize and Chrome cookies

Corion,
Here is the issue:
First, let's start with the html file that I have used to test the script (WMC.html)


<html>

<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title>Testing hyperlink file Downloads</title>
</head>

<body>
<h2>Testing Download of hyperlinked files using WWW::Mechanize::Chrome
+</h2>
<p></p>
Let's try downloading this <a href="/my_Files/csv_File.csv">CSV File</
+a>
<br/><br/>
 
</body>

</html>
[download]

Now here is the Perl script.


#!/usr/bin/perl -w

use Log::Log4perl qw(:easy);
use WWW::Mechanize;
use WWW::Mechanize::Chrome;
use strict;

my $cookie_dir = 'C:/Users/some_user/AppData/Local/Google/Chrome/User 
+Data/Default/';    #chrome cookies path 

#my $mech = WWW::Mechanize::Chrome->new( data_directory => $cookie_dir
+);
my $mech = WWW::Mechanize->new();

my $uri = URI->new( "https://www.your_site.com/WMC.html" ); 

$mech->get( $uri );
unless ($mech->success)
{
    my $mesg = $mech->response->status_line;
    print $mesg;
    goto FINISH;
}

my $path = "/path";        
my @links = $mech->find_all_links( url_regex => qr/\.csv/i );
my @urls = map { $_->[0] } @links;
for my $foo (@urls)
{
    my $filename = $path.$foo;

    $mech->get($foo, ':content_file'=>$filename);
    my $file_content  = $mech->get($foo);
    print $file_content->content();
}
            
print "Success\n";
FINISH :
[download]

When I use the WWW::Mechanize instance, the script runs fine. It prints and saves the file content to disk.
However, when the WWW::Mechanize::Chrome instance is used I get the following error message:

Cannot navigate to invalid URL

-32000 at C:/Perl/perl/site/lib/Chrome/DevToolsProtocol/Target.pm line
+ 490
[download]

Comment on Re^9: Need help with WWW::Mechanize and Chrome cookies Select or Download Code

Replies are listed 'Best First'.
Re^10: Need help with WWW::Mechanize and Chrome cookies by Corion (Patriarch) on Jul 11, 2021 at 15:20 UTC
`my @urls = map { $_->[0] } @links;` [download] That's not the documented way to get the URL from a WWW::Mechanize::Link object. You should use: `my @urls = map { $_->url_abs } @links;` [download]	[reply] [d/l] [select]
Re^10: Need help with WWW::Mechanize and Chrome cookies by Corion (Patriarch) on Jul 11, 2021 at 14:09 UTC
This is really weird - what is the value of the URL you're trying to navigate to when that error occurs? Where does that error occur?	[reply]
Re^11: Need help with WWW::Mechanize and Chrome cookies by bakiperl (Beadle) on Jul 11, 2021 at 14:42 UTC
I sent you the url. The error occurs on both of these calls: `$mech->get($foo, ':content_file'=>$filename); my $file_content = $mech->get($foo);` [download]	[reply] [d/l]
Re^12: Need help with WWW::Mechanize and Chrome cookies by Corion (Patriarch) on Jul 11, 2021 at 15:10 UTC
Yes. So what is the value of `$foo`? Did you print it? Is the loop necessary for the bug to happen, or does the error happen with the hardcoded value for `$foo` as well? From the URL you sent to me, it seems that WWW::Mechanize returns absolute URLs, while WWW::Mechanize::Chrome returns whatever URLs are contained in the file. Simple `print`ing of the values can confirm that to you. Until WWW::Mechanize::Chrome achieves API parity with WWW::Mechanize in that part, consider converting the URL to absolute yourself.	[reply] [d/l] [select]
Re^13: Need help with WWW::Mechanize and Chrome cookies by bakiperl (Beadle) on Jul 11, 2021 at 16:50 UTC
Re^14: Need help with WWW::Mechanize and Chrome cookies by Corion (Patriarch) on Jul 11, 2021 at 19:07 UTC
Some notes below your chosen depth have not been shown here


Clear questions and runnable code get the best and fastest answer
	PerlMonks