Re^3: chomp() is confusing

[Anonymous Monk]

As of today using Strawberry Perl 5.16.0.1 (64bit), Perl doesn't automatically adapt chomp at all to the platform nor converts automatically the carriage return from files that are read. For me, it always just remove \n, and that's all. My solution: either change the special var as suggested above, or use your own chomp with a substitution regexp: s/(\n|\r)//g Or of course, change the carriage return of all your files.

[Athanasius]

As of today using Strawberry Perl 5.16.0.1 (64bit), Perl doesn't automatically adapt chomp at all to the platform nor converts automatically the carriage return from files that are read.

But Perl doesn’t work like that. As ikegami explained in his post up-thread, removal of the carriage return (CR) character occurs before the string is ever fed to chomp. See the section Defaults and how to override them in PerlIO. On Windows platforms the IO layers default to unix (the lowest level layer) plus a crlf layer on top of this. This crlf is:

A layer that implements DOS/Windows like CRLF line endings. On read converts pairs of CR,LF to a single "\n" newline character. On write converts each "\n" to a CR,LF pair.

So, it is the crlf layer that converts each literal 0d0a to Perl’s logical newline "\n" (0a), and this happens as each line of the file is read in.

The input record separator, $/, defaults to "\n" (0a), Perl’s logical newline character, regardless of the platform. And chomp($string) removes from the end of $string the character(s) (if any) currently assigned to $/. Which all works out fine, because on Windows each CRLF has already been converted into a LF by the crlf IO layer before chomp comes into play.

Now, I can’t test Strawberry Perl 5.16.0.1 (64bit) as I don’t have a 64-bit OS. However, if the latest 64-bit version of Strawberry Perl does handle CRs incorrectly, it’s likely the difference is due to a change in the default IO layers. It’s unlikely to be in any way related to the implementation of the chomp function, for the reasons outlined above. I can confirm that the 32-bit versions of Strawberry Perl 5.14.2 and 5.16.0 (64int) treat carriage returns identically when reading in a text file under Vista.

Of course, the above applies only to text files. If binmode is applied to a file handle before it is read, conversion is suppressed, CRs are retained, and chomp has no effect on these CRs (unless $/ has been re-assigned).

Update: Try specifying the input layer explicitly:

open(my $fh, '<:crlf', $filename) or die ...

[Anonymous Monk]

Good news, as of Perl 5.1, you can use the \R $line =~ s/\R//g; It will figure it out.