Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Using pragma utf8::all in processing non-utf data.

by farang (Hermit)
on Sep 05, 2013 at 05:39 UTC ( #1052489=note: print w/ replies, xml ) Need Help??


in reply to Using pragma utf8::all in processing non-utf data.

If you have a file with a known encoding, for instance koi8-r, just use

open( $fh, '< :encoding(koi8-r)', "in_file");
and it should work fine under utf8::all.
And i want the scripts to: a) warn me when i open a file that contains non-utf characters rather than die at it;
I don't think it's possible to do it "when the file is opened" because the error arises when some non-Unicode utf8 sequence is read into Perl's internals. One way to do it is to use eval while reading the file line-by-line and trap the error. Here is some code which does that, trying first in utf8 and if that fails to be valid, warns and retries with koi8-r.
use strict; use warnings; use utf8::all; open(my $fh, '<', "in_file") or die "cannot open in_file: $!"; eval { process_file_by_line() }; if ( $@ =~ /does not map to Unicode/ ) { warn $@; print "...trying encoding koi8-r instead of utf8\n\n"; close $fh; open( $fh, '< :encoding(koi8-r)', "in_file") or die "cannot open i +n_file: $!"; process_file_by_line(); } elsif ( $@ ne '' ) { die $@; # bail out on other eval errors } sub process_file_by_line { while ( <$fh> ) { print; # whatever else... } }


Comment on Re: Using pragma utf8::all in processing non-utf data.
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1052489]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2014-09-17 02:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (56 votes), past polls