Re: Processing spreadsheet with some cells in ASCII, other cells in UTF-8

[stevieb]

This is my first ever attempt at non-ascii processing, so I'll let the more experienced Monks criticize if this is the wrong approach, or if there's a better one.

After a very quick dig online, I found that setting binmode on all the file handles can fix the issue:

Input file:

$ cat in.txt
}cýæu}*]…‘¦å
hello ›ÇÁ

Code:

use warnings;
use strict;

open my $fh, '<', 'in.txt'
  or die $!;

open my $wfh, '>', 'out.txt'
  or die $!;

binmode $fh, ":utf8";
binmode $wfh, ":utf8";
binmode STDOUT, ":utf8";

while (<$fh>){
    chomp;
    print $wfh "file: $_\n";
    print "stdout: $_\n";
}

Output:

# output file
file: }cýæu}*]…‘¦å
file: hello ›ÇÁ

# stdout
stdout: }cýæu}*]…‘¦å
stdout: hello ›ÇÁ

[Amphiaraus]

The problem in our team's Perl script was fixed simply by adding a one-line change at the top of the script:

binmode STDOUT, ':utf8'; #This handles the multiple encoding for language menus in the Perl IO layers.

The above one-line change fixed the problem, the Perl script can now process contents of Excel spreadsheet cells, no matter what type of character encoding was used in them.

No additional low-level function calls, to decode or encode, were needed.

This change was taken from one of the replies to my original question. Thanks for your help.

The various web pages found in the replies, which discussed character encoding, were also very helpful.