Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: buffering zipped pipes

by BrowserUk (Pope)
on Oct 04, 2016 at 08:07 UTC ( #1173218=note: print w/replies, xml ) Need Help??


in reply to buffering zipped pipes

I'd skip all the seeks and tells and use a thread and a queue this way:

#! perl -slw use strict; use threads; use Thread::Queue; my $file = $ARGV[ 0 ]; my $Q = new Thread::Queue; async { open LIBBIN, "gunzip -c $file |" or die $!; while( <LIBBIN> ) { $Q->enqueue( $_ ); sleep 1 while $Q->pending > 100; ## Arbitrary + limit to stop it from running away with memory } $Q->enqueue( undef ); ## close queue }->detach; ## Main code swaps $Q->dequeue for <LIBBIN> ... while( my $line = $Q->dequeue ) { if( some circumstance ) { unshift @{ $Q->{queue} }, $line; ## 'push' a li +ne back to the queue. } ## do other stuff here.. }

The only slightly 'trick' thing here is that I'm reaching inside the Queue object to gain access to the underlying array reference with $Q->{queue}, which the OO purists would have a hissy fit about, but to my mind one of the big advantages of Perl's simple native OO mechanism, is that I as a user can easily extend -- or in this case regain access to existing -- functionality without having to go cap in hand to the authors of the module or complicate things with OO complexity.

If the basic mechanism of having a separate thread doing the IO and a queue to buffer the data fits with your mindset, then you should also take a look at the "ADVANCED METHODS" in the Thread::Queue docs. For the most part I consider their addition to the queue module an abomination, but for the first time ever, I can see your application might find them useful.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1173218]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2020-04-06 03:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    The most amusing oxymoron is:
















    Results (36 votes). Check out past polls.

    Notices?