Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Problem with LWP::UserAgent

by Paradigma (Novice)
on Jun 23, 2019 at 11:47 UTC ( [id://11101751]=note: print w/replies, xml ) Need Help??


in reply to Re: Problem with LWP::UserAgent
in thread Problem with LWP::UserAgent

For example https://www.7digital.com/

By further investigation I discover that some webs send pages compressed by gzip and at least all of those make Perl crash. Don't know if I can somehow enable the support by any additional header attribute, what I tried doesn't work so far - HTTP::Headers->header('Accept-Encoding' => 'gzip')

There may be also problem with expired SSL certificate not handling by LWP::UserAgent->request()

In the meantime I'm fetching this web externally by cURL, but I'm not too comfortable with this as the content seems not parsed well. In any case I would prefer retrieving the web pages internally via Perl's module

Alternately I would accept a suggestion to different more robust Perl's framework for fetching pages

Replies are listed 'Best First'.
Re^3: Problem with LWP::UserAgent
by holli (Abbot) on Jun 23, 2019 at 12:26 UTC
    The code below works just fine for me. Try running it. It could be you are missing a dependency or something. What is the exact error message you get?
    use strict; use LWP::UserAgent; use HTTP::Headers; my $ua = LWP::UserAgent->new( 'agent' => 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537 +.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36' ); my $hdr = HTTP::Headers->new( 'Content-Type' => 'text/plain', 'Content-Length' => 0, ); my $url = "https://www.7digital.com/"; my $req = HTTP::Request->new(GET => $url, $hdr); my $res = $ua->request($req); if ($res->is_success) { print $res->decoded_content; } else { die $res->status_line; }


    holli

    You can lead your users to water, but alas, you cannot drown them.

      Weird, $ua->request() by me crashes. The error reported by Perl is none (the crash is uncontrolled). The system exception code c0000005 stands for memory access violation (attempt to write or read to/from invalid address). What's your Perl version and can you share your ssleay32.dll and libeay32.dll?

      Event: APPCRASH Application: perl.exe App version: 5.26.3.2603 Error module: perl526.dll Error module version: 0.0.0.0 Exception code: c0000005 Exception offset: 000000000013ada7 OS version: 6.3.9600.2.0.0.256.4
        Smells like a broken module installation. I'd reinstall Net::SSLeay and if that doesn't work reinstall Perl. See if the problem persists. FWIF
        D:\ENV>dir D:\ENV\strawberry-5.26.1\perl\vendor\lib\auto\Net\SSLeay\*. +dll Datenträger in Laufwerk D: ist DATA Volumeseriennummer: B802-0C3B Verzeichnis von D:\ENV\strawberry-5.26.1\perl\vendor\lib\auto\Net\SSL +eay 30.11.2017 06:12 411.136 SSLeay.xs.dll 1 Datei(en), 411.136 Bytes 0 Verzeichnis(se), 479.543.558.144 Bytes frei D:\ENV>perl -V Summary of my perl5 (revision 5 version 26 subversion 1) configuration +: Platform: osname=MSWin32 osvers=6.3 archname=MSWin32-x64-multi-thread uname='Win32 strawberry-perl 5.26.1.1 #1 Sun Sep 24 05:32:33 2017 +x64' config_args='undef' hint=recommended useposix=true d_sigaction=undef useithreads=define usemultiplicity=define use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -D__USE_MINGW_ANS +I_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLIC +IT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields' optimize='-s -O2' cppflags='-DWIN32' ccversion='' gccversion='7.1.0' gccosandvers='' intsize=4 longsize=4 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long long' ivsize=8 nvtype='double' nvsize=8 Off_t='long long' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='g++.exe' ldflags ='-s -L"D:\ENV\strawberry-5.26.1\perl\lib\CORE" -L"D:\ENV\ +strawberry-5.26.1\c\lib"' libpth=D:\ENV\strawberry-5.26.1\c\lib D:\ENV\strawberry-5.26.1\c\x +86_64-w64-mingw32\lib D:\ENV\strawberry-5.26.1\c\lib\gcc\x86_64-w64-m +ingw32\7.1.0 libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 + -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 - +lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomd +lg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_ +32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 libc= so=dll useshrplib=true libperl=libperl526.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs dlext=xs.dll d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags='-mdll -s -L"D:\ENV\strawberry-5.26.1\perl\lib\CORE" -L" +D:\ENV\strawberry-5.26.1\c\lib"' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under MSWin32 Compiled at Sep 24 2017 05:47:34 @INC: D:/ENV/strawberry-5.26.1/perl/site/lib/MSWin32-x64-multi-thread D:/ENV/strawberry-5.26.1/perl/site/lib D:/ENV/strawberry-5.26.1/perl/vendor/lib D:/ENV/strawberry-5.26.1/perl/lib


        holli

        You can lead your users to water, but alas, you cannot drown them.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11101751]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2024-03-29 05:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found