Suffering with buffering - how to wake 'less'

[benizi]

The only thing that generally annoys me about the 'less' command (other than trying to google it) is that I can't figure out how to get it to not stop the program on the input end of its pipe. Take the following example:

perl -lwe 'print and /000$/ and warn "HERE$_\n" for 1..1e6' | less

HERE1000 through HERE14000 get printed to STDERR before less stops perl. Unbuffering output (select((select($_),$|=1)[0]) for STDOUT, STDERR) doesn't change anything.

perl isn't the program with which I'm having trouble, but hopefully what works for perl will point me in the right direction.

[ikegami]

Printing to a full pipe causes the print to block. less doesn't read from its input until it needs to. I don't see any option to change that behaviour.

[benizi]

Nuts. I was hoping for something like +G or +F that wouldn't cause less to fully block.

Falling back to the ol' dosomething >& file &; less file approach.

Thanks as always ikegami.

[ikegami]

producer > file & less file ; rm file

doesn't work since less isn't expecting the file to change.

producer > file ; less file ; rm file

works, but less will only start up once the producer has ended.

[BrowserUk]

Try this:

#! perl -sw
use strict;
use threads;
use Thread::Queue;

my $Q = new Thread::Queue;

async{
    $Q->enqueue( $_ ) while <>;
    $Q->enqueue( undef );
}->detach;
sleep 1;

print while $_ = $Q->dequeue;

Name it buf(.pl) and use it like this:

perl -E"$_%1000 or say 'Here'. $_ for 1..1e6" | buf | less

[benizi]

Wow. Cool.

I was worried it wouldn't work, since your test wasn't testing the same thing as mine. (The warnings were a side channel indication of the fact that 'less' was stopping 'perl'). But, it seems to do the trick. Thanks a bundle.

...Except now I'm confused. How does it work? What is buf(.pl) doing that prevents 'less' from stopping it? or that forces 'less' to keep reading? I'm kind of back to my initial question from another angle: what is 'less' doing to 'perl' that indicates to 'perl' that it should pause its output?

[BrowserUk]

Once less has displayed the first screen full, it stops reading the pipe and waits for instructions from the keyboard. The pipe fills and won't accept anymore from the producer, so it blocks on the write until the pipe empties. Which it won't until you hit the space bar or similar.

buf.pl uses a shared queue as a buffer between two threads, The read thread just keeps reading and pushing to the queue. The write thread reads from that queue and writes to the pipe. Because the two are asynchronous, when the write thread blocks because the pipe to less is full, the reader just carries on filling up memory until the producer finishes.

The downside is that if the producer produces sufficient output, the perl process will eventually run out of memory and die.

[ikegami]

What is buf(.pl) doing that prevents 'less' from stopping it?

Nothing. buf's output thread is still blocking at the same spot as the producer would without buf.

less stops reading when it has enough to display what it needs to display. The producer doesn't know or care when that happens, so it keeps adding to the pipe until it fills up. Trying to add more to the pipe, the producer blocks until less starts emptying it.

buf reads all that it can as fast as it can. That's likely to be faster than the producer can produce output, so the pipe from the producer never fills, so the producer never blocks. buf keeps the lines it read in a Queue from where the writer will fetch them when it unblocks.

[ahmad]

You could try using another command, like:

perl -lwe 'print and /000$/ and warn "HERE$_\n" for 1..1e6' | tail -f

[benizi]

That omits the first 999,990 lines of that sample script. But, as used in my reply to ikegami, it's close to what I'm settling on. dosomething > file &; tail -f file Even though I really just want some 'read the rest of the input' command in 'less'.

[ikegami]

producer > file & tail -n +1 -f file | less ; rm file

Nah, no good, tail never exits, so less freezes trying to get the end of the file.