Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Getting/handling big files w/ perl

by karlgoethebier (Abbot)
on Nov 16, 2014 at 12:48 UTC ( [id://1107341]=note: print w/replies, xml ) Need Help??


in reply to Getting/handling big files w/ perl

HTTP::Range (OK, unmaintained for 10 years) as well as LWP::Parallel::UserAgent might be worth to take a look at.

Update: I just reread the curl manpage.

Curl has a --range option:

host:~ # curl --range 0-79 -o out.html perlmonks.org
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
+  Current
                                 Dload  Upload   Total   Spent    Left
+  Speed
  0    80    0    80    0     0    368      0 --:--:-- --:--:-- --:--:
+--     0
host:~ # wc -c out.html
80 out.html
host:~ # cat out.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

[download]

See also WWW::Curl

Regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Replies are listed 'Best First'.
Re^2: Getting/handling big files w/ perl
by Gisel (Novice) on Nov 17, 2014 at 07:00 UTC

    Karl—

    Actually I have been looking pretty hard at LWP::Parallel::UserAgent. It might be the tool I am looking for, but I am just not clear yet on how I would fit my problem into it.

    Also, this whole "information transmission" issue is not one I really have a firm footing on. The spin-up is taking me a while...

    Thanx for your input.

[reply]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://1107341]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others exploiting the Monastery: (6)
As of 2024-04-19 14:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found