http://www.perlmonks.org?node_id=868437


in reply to Chicanery Needed to Handle Unicode Text on Microsoft Windows

As far as I can see, the only reason you need :crlf is because you've specifically added the UNIX line ending (\n) to your output. It would be better to use the platform-independent $/. The :raw layer should preserve the line endings. So that reduces the chicanery somewhat.

Except for ASCII files, binmode($file_handle) was required on MSWin32 systems. :raw performs the same function so, while perhaps appearing to add to the chicanery, it certainly reduces the amount of code.

I don't have sufficient knowledge of UTF-16 to address that aspect of you post. What I would suggest is that, after removing :crlf and changing \n to $/, you try your test code without :perlio. You may still need it but it wouldn't hurt to check.

I agree there's a lot of Unicode-related documentation; however, everything I've made reference to is available here: PerlIO.

I ran a series of tests, click on Read more... to view.

Starting code:

#!perl use 5.12.0; use warnings; my $in_file = $^O eq 'MSWin32' ? 'utf16_LE_prob.dos_dat' : 'utf16_LE_prob.unix_dat'; my $out_file = $^O eq 'MSWin32' ? 'utf16_LE_prob.dos_out' : 'utf16_LE_prob.unix_out'; my $in_mode = $^O eq 'MSWin32' ? '<:raw' : '<'; my $out_mode = $^O eq 'MSWin32' ? '>:raw' : '>'; open my $in_fh, $in_mode, $in_file or die $!; open my $out_fh, $out_mode, $out_file or die $!; while (my $line = <$in_fh>) { print $out_fh $line; } close $out_fh; close $in_fh;

Input files in UNIX and DOS formats:

$ cat -vet utf16_LE_prob.unix_dat utf16_LE_prob.dos_dat Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$

Output after running on UNIX platform:

$ cat -vet utf16_LE_prob.unix_out Line 1$ Line 2$ $

Output after running on DOS platform:

$ cat -vet utf16_LE_prob.dos_out Line 1^M$ Line 2^M$ ^M$

Changing the while loop to chomp input and add $/ (not \n) to output:

while (my $line = <$in_fh>) { chomp $line, print $out_fh $line, $/; }

New output:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$

Adding :crlf to MSWin32 input and output modes (now = :raw:crlf) and there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$

With :raw:perlio:crlf, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$

And, for completeness, with :raw:perlio, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$

-- Ken