Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Redirecting/Restoring of Memory Files

by ELISHEVA (Prior)
on Jun 25, 2015 at 11:43 UTC ( #1131938=perlquestion: print w/replies, xml ) Need Help??
ELISHEVA has asked for the wisdom of the Perl Monks concerning the following question:

Is there a better way to do this? Am I using the correct idiom? Is this a bug? My Perl version is 5.18 (Mac OS Yosemite system perl)

I recently noticed that attempts to dup/restore a file handle pointing to string turns that file handle into a black hole. Writes (e.g.print) to file handle don't seem to appear in the output when the file handle is restored after dupping. New input isn't appended to the string, nor does it replace the beginning of the re-opened memory file/string.

The black hole goes away if I reopen using substr($s,length($s)) where $s is the original variable used to create the memory file handle.

Trying to reopen in append mode also results in a black hole on my macbook

Below (hidden with readmore) is a script demonstrating the problem.

use strict; use warnings; use Config; use IO::String; sub demo { my ($option) = @_; my ($s1, $s2) = ('',''); my $sFormat = "s1='%s' s2='%s'\n"; my $fh1; print STDERR "===========\n"; open($fh1, '>', \$s1) or die "Can't open fh1"; print $fh1 'Hi Thing One'; printf STDERR $sFormat, $s1, $s2; # => s1='Hi Thing One' s2='' open(my $fhSave, '>&', $fh1) or die "Can't save fh1"; close $fh1; open($fh1, '>', \$s2) or die "Can't redirect fh1"; print $fh1 'Hi Thing Two*Bye Thing Two'; printf STDERR $sFormat, $s1, $s2; # => s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' print STDERR "Option $option"; if ($option == 1) { print STDERR " (opening in >& mode)\n"; open($fh1, '>&', $fhSave) or die "Can't restore fh1"; } elsif ($option == 2) { print STDERR " (opening on a substr)\n"; open($fh1, '>', \ substr($s1, length($s1)) ) or die "Can't restore fh1"; } elsif ($option == 3) { print STDERR " (opening in >> mode)\n"; open($fh1, '>>', $fhSave ) or die "Can't restore fh1"; } print $fh1 '*Bye Thing One'; printf STDERR $sFormat, $s1, $s2; # OUTPUT FROM OPEN OPTIONS #1# => s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' #2# => s1='Hi Thing One*Bye Thing One' s2='Hi Thing Two*Bye Thing Tw +o' #3# => s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' } print STDERR "\n".qq!Perl $^V on $^O arch $Config{archname}\n!; demo(1); demo(2); demo(3); print "\n"; exit(0);

The output on my macbook is

Perl v5.18.2 on darwin arch darwin-thread-multi-2level =========== s1='Hi Thing One' s2='' s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' Option 1 (opening in >& mode) s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' =========== s1='Hi Thing One' s2='' s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' Option 2 (opening on a substr) s1='Hi Thing One*Bye Thing One' s2='Hi Thing Two*Bye Thing Two' =========== s1='Hi Thing One' s2='' s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two' Option 3 (opening in >> mode) s1='Hi Thing One' s2='Hi Thing Two*Bye Thing Two'

Edit1: clarified instructions for running script, more details about Perl version. Thank-you Disciplus for suggestions.

Edit 2: revised script so that it just runs without needing to uncomment anything and also prints out your perl version - once again thank-you Disciplus, this time for the idea of using config.

Edit 3: revised script to include option to reopen file in append mode - which also doesn't work on Perl 5.18 on Darwin

Replies are listed 'Best First'.
Re: Redirecting/Restoring of Memory Files
by Discipulus (Monsignor) on Jun 25, 2015 at 12:36 UTC
    the only thing i can say is that your (sligthly modified) demo will output your same output in modern Strwaberry release,
    but never print '*Bye Thing One' in a old 5.8 release:

    L*
    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
Re: Redirecting/Restoring of Memory Files
by flexvault (Monsignor) on Jun 25, 2015 at 15:16 UTC

    Hello ELISHEVA,

    I'm not sure I understand what you are trying to do. Your description implies to me that you are trying to save a file handle for later use, but your trying to use I/O to a scalar in memory.

    my ($s1, $s2) = ('',''); . . . open(my $fh1, '>', \$s1) or die "Can't open fh1";
    You are opening a defined but empty string variable by passing a reference to the string. This maybe a new feature of Perl that I'm not aware of, but usually you open a file on an external media.

    More clarification will be needed to give a better answer, but if you want to save the file handle ( $fh1 ) for later use, you could use a hash or array, which will preserve the definitions of $fh1.

    Regards...Ed

    "Well done is better than well said." - Benjamin Franklin

      Perl does have a feature of being able to open a file handle/stream on a string buffer rather than external media. There are several reasons why it is sometimes useful to treat a string as if it were a file. Three reasons that come to mind:

      • testing - you want to verify that a subroutine that normally sends output to STDERR , STDOUT, or even a file is in fact sending the expected output. By temporarily redirecting the stream to a string, you can examine and test the output more easily
      • simplification of API/maintenance - you are writing a program that needs to work with both file streams and data stored in strings and want one API/programming metaphor for both - either because it makes the API easier to learn or because you simply don't want to write two versions of code, one for strings and one for data read in from a file.
      • performance tuning - trading memory for speed for temp files. Read/write access on strings is obviously more memory intensive than storing the same data in a file. On the other hand, read/writes to disk are sometimes slower. Being able to switch between writing to an actual temp file vs. string and switch easily between the two makes benchmarking and performance tuning easier.

      For more information on this feature of Perl, see open and search for the phrase "Perl scalars". I believe support has been part of the Perl core since 5.8 (which is why I presume Disciplus tested that version - which is really old). See also IO::Scalar and IO::String. IO::Scalar can be used to bring support for in memory/scalar backed file handles to even earlier versions of Perl.

        Thank you ELISHEVA,

        It's always good to learn something new, and doing I/O to a string is new to me. Since I use 'diff' a lot in *nix and also have my own editor for working with development files, I'm not sure how often I would use it.

        As far as your 'black hole', you'll see the same with a regular file. I have prematurely closed a file handle and then later added a 'print' statement. No error, no warning and no data. Also if you save a file handle to disk and then retrieve, you need some 'Perl magic' to get it to be valid file handle again.

        Thanks for the updated information...Ed

        Regards...Ed

        "Well done is better than well said." - Benjamin Franklin

Re: Redirecting/Restoring of Memory Files
by Anonymous Monk on Jun 26, 2015 at 06:21 UTC
    Well, perldoc does say:
    You may also, in the Bourne shell tradition, specify an EXPR beginning with ">&", in which case the rest of the string is interpreted as the name of a filehandle (or file descriptor, if numeric) to be duped (as dup(2))
    And the UNIX system call dup works only with descriptors (files, pipes, FIFOs, sockets etc). So you probably shouldn't expect it to work with Perl data without an underlying descriptor.
    The black hole goes away if I reopen using substr($s,length($s)) where $s is the original variable used to create the memory file handle.
    Or if you reopen in append mode, as in
    } else { open($fh1, '>>', \$s1) or die "Can't restore fh1"; }

      Your success with append mode might be Perl version/system dependent. On my macbook, I still get the black hole when I reopen in append mode. I've revised the script in the original post to also demonstrate what happens when one re-opens in append mode.

      I'm also not comfortable with the idea that "one shouldn't expect" a file handle to work with dup. As far as I know, Perl doesn't give me a way to tell if a file handle was created on a string.There is no way to gracefully degrade or warn the user if they hand my subroutine a file handle and it just happens to be backed with a string.

        Effectively i had no success in append mode (all tests on windows as above) as suggested by Anonymous.

        The possibility to open an handle to a string is in the Perl Faq and also the possibility to dup a filehandle is in the Perl Faq. Also, as you already said, is worth to read open and search for the phrase "Perl scalars".

        The crucial point seems to be spotted by Anonymous here above: the '>&' or '>&=' trick works only with descriptors and in your case ELISHEVA we have no descriptor:
        # perl -e ' open $h,">", \$s or die; print qq(fileno: [).fileno ($h).q +q(]\n);' fileno: [-1]
        And fileno doc says it explicitly:
        Returns the file descriptor for a filehandle, or undefined if the filehandle is not open. If there is no real file descriptor at the OS level, as can happen with filehandles connected to memory objects via open with a reference for the third argument, -1 is returned.
        So it seems only an hedge case worth to be inserted in the open documentation and in the relevant Faq

        As side note i remember the $^F or maximum file descriptor. How an handle opened on a reference acts in respect of this? is close_on_exec set in the right way? or this will result in zombie handles?
        Might be worth to clarify also this in the docs.

        L*
        There are no rules, there are no thumbs..
        Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
        Your success with append mode might be Perl version/system dependent. On my macbook, I still get the black hole when I reopen in append mode.
        I mean
        open($fh1, '>', \ substr($s1, length($s1)) ) or die "Can't restore fh1";
        replace '>' with '>>' and \ substr($s1, length($s1)) with \$s1 and you'll get
        open($fh1, '>>', \$s1) or die "Can't restore fh1";
        I'm also not comfortable with the idea that "one shouldn't expect" a file handle to work with dup. As far as I know, Perl doesn't give me a way to tell if a file handle was created on a string.There is no way to gracefully degrade or warn the user if they hand my subroutine a file handle and it just happens to be backed with a string.
        Well I'm not saying that open is being reasonable here. IMO, yes, it should work, or at least warn if it doesn't. That being said, there is a way to tell the difference
        Returns the file descriptor for a filehandle, or undefined if the filehandle is not open. If there is no real file descriptor at the OS level, as can happen with filehandles connected to memory objects via "open" with a reference for the third argument, -1 is returned. - fileno
Re: Redirecting/Restoring of Memory Files
by clueless newbie (Chaplain) on Jun 26, 2015 at 12:02 UTC
    Hope you and yours are well --- we missed you.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1131938]
Approved by Discipulus
Front-paged by Discipulus
help
Chatterbox?
and the rats come out to play...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2018-06-18 08:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?



    Results (109 votes). Check out past polls.

    Notices?