How to make an HTTP request with an equivalent of curl's --max-time?

Cody Fendant has asked for the wisdom of the Perl Monks concerning the following question:

I'm grabbing data from a special Twitter firehose-type URL.

It will send information indefinitely*, so in order to manage the amount of data received by the script, it's recommended to use the maximum-time option.

Here's what I'm doing, a system() call containing:

/usr/bin/curl 
--request 'GET' 'https://stream.twitter.com/1.1/statuses/sample.json' 
--max-time 180 
--header 'Authorization: $header'
[download]

Where $header is some OAuth magic.

So, is it possible to do this without the shell-out to curl and keep it all within Perl?

As far as I can see, that maximum time option isn't directly related to HTTP Headers, it's curl-specific.

* Updated to add documentation on this "indefinitely" thing. From the Twitter documentation::

To connect to the Streaming API, form a HTTP request and consume the resulting stream for as long as is practical. Our servers will hold the connection open indefinitely, barring server-side error, excessive client-side lag, network hiccups, routine server maintenance or duplicate logins.

Comment on How to make an HTTP request with an equivalent of curl's --max-time? Select or Download Code

Replies are listed 'Best First'.
Re: How to make an HTTP request with an equivalent of curl's --max-time? by atcroft (Abbot) on Jan 14, 2015 at 06:11 UTC
My first thought would be to wrap the call in an eval{} block, and set an alarm(), similar to the docs for the alarm function. Hope that helps.	[reply]
Re: How to make an HTTP request with an equivalent of curl's --max-time? by blindluke (Hermit) on Jan 14, 2015 at 06:51 UTC
Take a look at Time::Limit. It is possible that it will be the right tool for the task ahead of you. See: Self terminating a script after a specified time for a problem similar to yours. Take note of this anonymous reply with some very useful notes about alarm. - Luke	[reply]
Re: How to make an HTTP request with an equivalent of curl's --max-time? by Anonymous Monk on Jan 14, 2015 at 09:01 UTC
Use :content_cb option, and inside it perform a max-time check ... lwp has built in max-size check	[reply]
Re: How to make an HTTP request with an equivalent of curl's --max-time? by Anonymous Monk on Jan 14, 2015 at 07:09 UTC
Where is the documentation for this thing?	[reply]
Re^2: How to make an HTTP request with an equivalent of curl's --max-time? by blindluke (Hermit) on Jan 14, 2015 at 07:27 UTC
For the `max-time` option? It's in the manpage for curl. `$ man curl (...) -m, --max-time <seconds> Maximum time in seconds that you allow the whole operation to ta +ke. This is useful for preventing your batch jobs from hanging for h +ours due to slow networks or links going down. Since 7.32.0, this option accepts decimal values, but the actual timeout will +decrease in accuracy as the specified timeout increases in decimal precis +ion. See also the --connect-timeout option. If this option is used several times, the last one will be used. (...)` [download] - Luke	[reply] [d/l] [select]
Re^3: How to make an HTTP request with an equivalent of curl's --max-time? by Anonymous Monk on Jan 14, 2015 at 08:54 UTC
For the max-time option? It's in the manpage for curl. No, not for the self-explanatory option, for the magic url that is immune to regular HTTP GET, that it needs a client induced timeout	[reply]
Re^4: How to make an HTTP request with an equivalent of curl's --max-time? by blindluke (Hermit) on Jan 14, 2015 at 09:16 UTC
Re^5: How to make an HTTP request with an equivalent of curl's --max-time? by Anonymous Monk on Jan 14, 2015 at 09:20 UTC
Some notes below your chosen depth have not been shown here
Re^4: How to make an HTTP request with an equivalent of curl's --max-time? by Cody Fendant (Hermit) on Jan 14, 2015 at 22:13 UTC


There's more than one way to do things
	PerlMonks