Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Autoflush and web browsers (was: Clever autoflush detail)

by talexb (Canon)
on Jun 20, 2007 at 17:49 UTC ( #622339=perlquestion: print w/replies, xml ) Need Help??

talexb has asked for the wisdom of the Perl Monks concerning the following question:

I received a query from a client about how to get autoflush working, so cobbled together a simple example. First, I did it in shell,

#!/bin/sh echo "Content-type: text/plain"; echo for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do echo "Value is $i"; sleep 1; done

This worked fine (it output a line every second), and I translated it into Perl.

#!/usr/bin/perl -w $|=1; use CGI qw/:standard/; { my $cgi = CGI->new; print $cgi->header('text/plain'); print "Autoflush is currently ".($|?"on":"off").".\n"; for ( 1..5 ) { print "Value is $_.\n"; sleep 1; } print "Autoflush is currently ".($|?"on":"off").".\n"; }

This didn't work. Nothing was output until the script finished. I was stumped. I search Perlmonks and found lots of information on autoflush questions, nodes Autoflush $|?, Autoflushing Laser printer, How to get explicity the autoflush var ($|) of a TIED HANDLER?, Autoflushing revisited., Simple autoflush module, OUTPUT_AUTOFLUSH, autoflush and 'DESTROY', autoflushing.. and When to use Autoflush. Nothing solved my problem.

Finally, over on IRC, I put my question and got a response from Caelum -- use PRE tags.

#!/usr/bin/perl -w $|=1; use CGI qw/:standard/; { my $cgi = CGI->new; print $cgi->header('text/html'); print pre("Autoflush is currently " . ($|?"on":"off") ); for ( 1..5 ) { print pre("Value is $_.</pre>"); sleep 1; } print pre("Autoflush is currently " . ($|?"on":"off") ); }

Well, to cut a long story short, it works. But why?

Shouldn't the autoflushing (happening down at layer 2 or 3) be immune to the presentation (up at layer 7)? Is this a Perl oddity?

What is going on?

Update: As fenLisesi points out, the closing pre tag in the 'Value is' print statement shouldn't be there, but I won't edit the original node to reflect that..

Update 2: I don't mind changing the title of this node to Autoflush and web browsers, but that's naming the thread after the fact -- after we know (or suspect) that it's the interaction with the browser that's causing the odd behaviour. In addition, it looks like most of the discussion on this thread is over (at the time of writing, the most recent reply to the root was shmem's dated 18 hours ago), so this consideration may not be necessary.

Update 3: Brother benizi makes a very good point -- retitling the node would be an aid to future searches, therefore it's a Good Thing. I've held off voting on the consideration or making a title change, but now I'm convinced.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Replies are listed 'Best First'.
Re: Clever autoflush detail
by shmem (Chancellor) on Jun 20, 2007 at 21:12 UTC
    It's not a CGI thing - it's the charset that CGI adds.
    print "Content-type: text/plain;\r\n\r\n";

    and

    print "Content-type: text/plain; charset=ISO-8859-1\r\n\r\n";

    give different results - no matter which of shell, perl, with or without CGI is involved. For the latter, the browser seems to gather all its input first read in blocks, presumably to apply conversions on the complete text to each. (tested with Firefox only).

    Try appending "\0" x 1024 to each line using CGI with print $cgi->header("text/plain").

    Another way to confirm that unbuffered IO is turned on with CGI is snarfing the packets with eg wireshark.

    ...another update: even the trailing ; after text/plain makes a difference. If you are not outputting that, the browser buffers.

    --shmem

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
Re: Clever autoflush detail
by BrowserUk (Pope) on Jun 20, 2007 at 18:46 UTC

    I don't do cgi, but surely the problem (being fixed by using PRE tags), is not at the output end, but at the browser end?

    Normally, html ignores CR/LF. So, the browser cannot display the line until it gets something (like: BR or P) that tells it that this 'line' is complete and can be formatted. Using the PRE tags means that CR/LF is respected, so the browser can display the line as soon as it sees those characters.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
        Normally, html ignores CR/LF. So, the browser cannot display the line until it gets something (like: BR or P) that tells it that this 'line' is complete and can be formatted.

      Right, but I made a point of setting the content type to be text/plain (that is, specifically not to be HTML), and that worked correctly using the shell script, but not using the Perl script. That's why I'm confused.

      Alex / talexb / Toronto

      "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Re: Clever autoflush detail
by Joost (Canon) on Jun 20, 2007 at 18:44 UTC
    my guess is that browsers cache text content until a tag is filled or until some kind of threshold is reached so that they don't have to do a lot of reformatting while downloading content (mozilla seems especially fond of doing this, by the way).

    Since pre tags have predictable formatting you'll see the content "right away". Adding some other tags around each line, like <br> or <p> will probably work too.

Re: Clever autoflush detail
by blahblah (Friar) on Jun 20, 2007 at 19:09 UTC
    Its something in CGI.pm.

    To wit:

    #!/usr/bin/perl -w $|++; { print "Content-Type:text/html\n\n"; print "Autoflush is currently ".($|?"on":"off").".\n"; for ( 1..5 ) { print "Value is $_.\n"; sleep 1; } print "Autoflush is currently ".($|?"on":"off").".\n"; }

    I don't have time to look exactly - but likely caching or something else. Take a look at the raw headers. blahblah
Re: Clever autoflush detail
by eric256 (Parson) on Jun 20, 2007 at 19:52 UTC

    Changing just the content type in your first makes it work as well with no pre tags.

    In fact if i try your sh script with text/plain it waits and shows them all together, but if i change it to text/html it displays them immediatly. So in both cases it only buffers/caches text/plain for me and text/html works. I would guess then that the problem lies either with the webserver or the web browser.


    ___________
    Eric Hodges

      Well, I tried three of the combinations of (using CGI|not using CGI) and content type (text/html|text/plain):

      CGI no CGI
      text/html Worked fine using PRE tags. (not tried)
      text/plain Did not work -- buffered all output!! Worked fine.

      I had a look at the CGI source code, and there didn't seem to be anything that talked about ignoring $|, but I will admit I only had a quick look at this Serious Collection of Magic.

      In any case, I have a workaround that works (no CGI, text/plain content type) and I'll communicate that back to my client.

      But if someone can find out if this deep magic really is happening in CGI, I'd be fascinated to hear about it.

      Alex / talexb / Toronto

      "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

[OT] Re: Clever autoflush detail ... or ... Autoflush and web browsers
by benizi (Hermit) on Jun 25, 2007 at 02:56 UTC

    Re: Update 2. The point of retitling the node is, from my understanding, to make it easier for people searching later on. Even if the discussion has concluded, if I were looking for information on autoflush, the new title would be more useful. If I'm in a situation where the browser might be the problem, the 'browser' in the title can clue me in that the thread is probably useful. And, if my problem has nothing to do with browsers, it's even more important; I'd much rather see "Autoflush and web browsers" instead of the rather vague "Clever autoflush detail", so I could then skip that Super Search result.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://622339]
Approved by ww
Front-paged by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2019-10-22 21:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?