Is there a way to open a memory file with binmode :raw?

[stevieb]

Instead of using a temporary disk file, I thought I'd try to use a memory file (scalar ref), but I need to be able to use the handle with binmode $fh, ':raw' (or equivalent). Is this possible?

use strict;
use warnings;

my $mf;
my $fh;

# mem write

open $fh, '>', \$mf or die $!;
print $fh "hello\n";
close $fh;

# mem read

open $fh, '<', \$mf or die $!;
binmode $fh, ':raw';

while (<$fh>){
    print "mem: " . (unpack "H*", $_) . "\n";
}
close $fh;

# file write

open $fh, '>', 'file.txt' or die $!;
print $fh "hello\n";
close $fh;

# file read

open $fh, '<', 'file.txt' or die $!;
binmode $fh, ':raw';

while (<$fh>){
    print "file: " . (unpack "H*", $_) . "\n";
}

Output... note the difference after the 'f'. I need the mem file to reflect that of :raw like the value for file.

mem:  68656c6c6f0a
file: 68656c6c6f0d0a

[kcott]

G'day stevieb,

I don't have MSWin available, but I can fake it sufficiently for this test with:

use open IO => ':crlf';

Here's four ways to do what you want (plus a fifth just to show what happens when one of those isn't used). There may, of course, be other ways I didn't think of.

#!/usr/bin/env perl -l

use strict;
use warnings;
use autodie;
use open IO => ':crlf';

my ($f, $fh); 
my $mf = \$f;
my $test_file = 'pm_1144333_test_file.txt';

open $fh, '>', $mf;
print $fh 'hello';
close $fh;
print_raw_1('mem:  ', $mf);
print_raw_2('mem:  ', $mf);
print_raw_3('mem:  ', $mf);
print_raw_4('mem:  ', $mf);
print_raw_5('mem:  ', $mf);

open $fh, '>', $test_file;
print $fh 'hello';
close $fh;
print_raw_1('file: ', $test_file);
print_raw_2('file: ', $test_file);
print_raw_3('file: ', $test_file);
print_raw_4('file: ', $test_file);
print_raw_5('file: ', $test_file);


sub print_raw_1 {
    my ($prompt, $file) = @_;

    open my $fh, '<:raw', $file;
    print '1. ', $prompt, unpack 'H*' while (<$fh>);
    close $fh;
}

sub print_raw_2 {
    my ($prompt, $file) = @_;

    open my $fh, '<', $file;
    binmode $fh, ':raw';
    print '2. ', $prompt, unpack 'H*' while (<$fh>);
    close $fh;
}

sub print_raw_3 {
    my ($prompt, $file) = @_;

    use open IN => ':raw';
    open my $fh, '<', $file;
    print '3. ', $prompt, unpack 'H*' while (<$fh>);
    close $fh;
}

sub print_raw_4 {
    my ($prompt, $file) = @_;

    use open IO => ':raw';
    open my $fh, '<', $file;
    print '4. ', $prompt, unpack 'H*' while (<$fh>);
    close $fh;
}

sub print_raw_5 {
    my ($prompt, $file) = @_;

    open my $fh, '<', $file;
    print '5. ', $prompt, unpack 'H*' while (<$fh>);
    close $fh;
}

Here's the output:

1. mem:  68656c6c6f0d0a
2. mem:  68656c6c6f0d0a
3. mem:  68656c6c6f0d0a
4. mem:  68656c6c6f0d0a
5. mem:  68656c6c6f0a
1. file: 68656c6c6f0d0a
2. file: 68656c6c6f0d0a
3. file: 68656c6c6f0d0a
4. file: 68656c6c6f0d0a
5. file: 68656c6c6f0a

And, if I hadn't "faked it", i.e. not including the use open IO => ':crlf'; line, I just get:

1. mem:  68656c6c6f0a
2. mem:  68656c6c6f0a
3. mem:  68656c6c6f0a
4. mem:  68656c6c6f0a
5. mem:  68656c6c6f0a
1. file: 68656c6c6f0a
2. file: 68656c6c6f0a
3. file: 68656c6c6f0a
4. file: 68656c6c6f0a
5. file: 68656c6c6f0a

See also:

• the open function
• the open pragma
• PerlIO

and, if you're interested in internals:

• perliol - C API for Perl's implementation of IO in Layers
• perlapio - perl's IO abstraction interface

[stevieb]

This is very informative kcott, thanks :)

However, it doesn't help me understand why on Windows, when printing to a file-based file handle, an \n is printed by default as \r\n into the file (without any binmode or IO trickery, it just does so naturally.

However, the default record separator \r\n is not printed to a memory file based handle, it is printed only as \n. I would expect that regardless of type of handle, the default OS record separator would be used. I can't find anywhere that states this discrepancy between a real file and printing the exact same thing to a scalar reference acting as a file handle.

That, or I'm missing something very basic.

[tye]

The problem is only with your expectations. Did you know that, in Unix, writing "\n" also becomes "\r\n", by default, just not in ordinary files. For example, it does that when writing to a TTY (having the default configuration).

Writing to a Perl scalar is not handled by the Windows clib, obviously. So there is no requirement that such writes emulate the default behavior of Windows' clib.

"\r\n" is the default text record separator for Windows text files.

[BrowserUk]

CRLF translation is feature of the Windows file systems; not the Perl language. The PerlIO layers emulate it when writing to the Windows file system.

One fairly typical usage of memory files is to reduce IO overheads by accumulating lines together into a single scalar and then write the entire file in one go.

If Perl applied the CRLF translation when writing to the memory file; then when the scalar is written to the file, the file system (or file system emulation) would apply the CRLF translation a second time and you would end up with \r\r\n.

Of course that could be avoided by applying the non default :raw layer to the actual output file; or by applying binmode; but that means extra steps are required.

Better to only apply CRLF translations when actually writing to actual file system files and then the default behaviours work together to produce the desired result.

[kcott]

Update: The accuracy of the information I linked to (perlport: Newlines) is in question. See tye's response to this node.

In Perl, \n is a logical newline. It does not necessarily represent the single ASCII character whose decimal value is 10.

Perhaps a read of "perlport: Newlines" will help clarify the situation for you.

[choroba]

Are you on MSWin? When writing to the file, you didn't change the binmode, so you're writing with :crlf. Line 26 should be

binmode $fh, ':raw';

[stevieb]

Thanks choroba.

Yes, I'm on MSWin for this test, and I want both CRLF to be written to the file, and then displayed after the fact. This does as expected with the actual file, but not the scalar ref memory pseudo file.

[choroba]

Sorry, I'm on Linux now. So do it the other way round: when printing to the scalar ref, set the binmode to :crlf.

[nikosv]

You might be interested in perl has built-in temp files , a subtle way of creating temporary filehandles

[stevieb]

That is genius! Thank you so much for this :)

I appreciate everyone's patience with me on this over the last month or so. Digging through all the docs was informative, but just a tad overwhelming trying to piece it all together which led to confusion.

When it was pointed out that a scalar ref is NOT a file and the two are treated in different ways helped a lot.

In this round, my goal was an anonymous temp file for which I had gone on to use File::Temp which I felt was a bit heavy-handed. I'm very surprised I never came across the link you shared in my travels.

Cheers everyone,

-stevieb

[tye]

Yeah, but that way of creating temporary files doesn't even honor $ENV{TMPDIR} (at least in some configurations), which makes it one of the worst ways to make temporary files in Perl in my book. I was rather shocked when I discovered such in the Perl source code (trying to figure out why Plack temporary files were being put into the wrong directory).