Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: LWP::UserAgent & memory problems

by daxim (Curate)
on Oct 23, 2012 at 17:08 UTC ( [id://1000502]=note: print w/replies, xml ) Need Help??


in reply to LWP::UserAgent & memory problems

What happens when you use the method mirror?

Replies are listed 'Best First'.
Re^2: LWP::UserAgent & memory problems
by Uree (Acolyte) on Oct 24, 2012 at 08:22 UTC

    Tried that one too. Happens exactly the same.

    Although, my approach to the issue was that, having tested a few different ways without success, it would actually be me doing something wrong/missing something rather than library's option/func not working as expected.
    For that reason, I stopped exploring different options and focused on spotting what Im missing out.

Re^2: LWP::UserAgent & memory problems
by Anonymous Monk on Oct 24, 2012 at 07:18 UTC

    It doesn't happen for me, which version of LWP are you using?

    $ perl -MDevel::VersionDump -MLWP -e 1
    Perl version: v5.14.1 on MSWin32
    Carp               -    1.26
    Config             - Unknown
    Devel::VersionDump -    0.02
    Exporter           -    5.66
    Exporter::Heavy    -    5.66
    Fcntl              -    1.11
    HTTP::Date         -    6.02
    HTTP::Headers      -    6.00
    HTTP::Message      -    6.03
    HTTP::Request      -    6.00
    HTTP::Response     -    6.03
    HTTP::Status       -    6.03
    LWP                -    6.04
    LWP::MemberMixin   - Unknown
    LWP::Protocol      -    6.00
    LWP::UserAgent     -    6.04
    Storable           -    2.30
    Time::Local        -  1.2300
    URI                -    1.60
    URI::Escape        -    3.31
    XSLoader           -    0.15
    constant           -    1.21
    overload           -    1.13
    strict             -    1.04
    vars               -    1.02
    warnings           -    1.12
    warnings::register -    1.02
    
    
      l$ perl -MDevel::VersionDump -MLWP -e 1 Perl version: v5.14.2 on linux (BREWED) Carp - 1.26 Config - Unknown Devel::VersionDump - 0.02 Exporter - 5.67 Exporter::Heavy - 5.67 Fcntl - 1.11 HTTP::Date - 6.02 HTTP::Headers - 6.05 HTTP::Message - 6.06 HTTP::Request - 6.00 HTTP::Response - 6.04 HTTP::Status - 6.03 LWP - 6.04 LWP::MemberMixin - Unknown LWP::Protocol - 6.00 LWP::UserAgent - 6.04 Storable - 2.39 Time::Local - 1.2300 URI - 1.60 URI::Escape - 3.31 XSLoader - 0.16 constant - 1.21 overload - 1.13 strict - 1.04 vars - 1.02 warnings - 1.12 warnings::register - 1.02

      So, you can actually execute that piece of code to dl a large file (~100MB+) and have it copied directly to a file, without using a portion of memory directly proportional to file's size?
      If that is the case, then I most definitely must be missing something. Can you spot something wrong with my code?

      PS: Didnt know 'Devel::VersionDump'; thanks for the input.

        So, you can actually execute that piece of code to dl a large file (~100MB+) and have it copied directly to a file, without using a portion of memory directly proportional to file's size?

        I did 38M/135M and , the memory never grew past 10M. For server (never past 12M) I used

         plackup -l localhost:80 -e " use Plack::App::Directory; my $app = Plack::App::Directory->new({ root => q[.] })->to_app; "

        Can you spot something wrong with my code?

        Nope, I don't see any mistakes but on linux (BREWED) makes me think maybe you're using a fudgy malloc? Kinda unlikely but I've seen folks complain about it :) I'm on win32 so you probably can't compare my Config:)

        $ perl "-V:.{0,}alloc.{0,}" d_attribute_malloc='undef'; d_malloc_good_size='undef'; d_malloc_size='undef'; d_mymalloc='undef'; i_malloc='define'; i_mallocmalloc='undef'; mallocobj='malloc.o'; mallocsrc='malloc.c'; malloctype='void *'; usemallocwrap='define'; usemymalloc='n';

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1000502]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (3)
As of 2024-03-28 16:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found