Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: Using pragma utf8::all in processing non-utf data.

by farang (Chaplain)
on Sep 05, 2013 at 05:39 UTC ( #1052489=note: print w/ replies, xml ) Need Help??

in reply to Using pragma utf8::all in processing non-utf data.

If you have a file with a known encoding, for instance koi8-r, just use

open( $fh, '< :encoding(koi8-r)', "in_file");
and it should work fine under utf8::all.
And i want the scripts to: a) warn me when i open a file that contains non-utf characters rather than die at it;
I don't think it's possible to do it "when the file is opened" because the error arises when some non-Unicode utf8 sequence is read into Perl's internals. One way to do it is to use eval while reading the file line-by-line and trap the error. Here is some code which does that, trying first in utf8 and if that fails to be valid, warns and retries with koi8-r.
use strict; use warnings; use utf8::all; open(my $fh, '<', "in_file") or die "cannot open in_file: $!"; eval { process_file_by_line() }; if ( $@ =~ /does not map to Unicode/ ) { warn $@; print "...trying encoding koi8-r instead of utf8\n\n"; close $fh; open( $fh, '< :encoding(koi8-r)', "in_file") or die "cannot open i +n_file: $!"; process_file_by_line(); } elsif ( $@ ne '' ) { die $@; # bail out on other eval errors } sub process_file_by_line { while ( <$fh> ) { print; # whatever else... } }

Comment on Re: Using pragma utf8::all in processing non-utf data.
Select or Download Code

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1052489]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (2)
As of 2015-11-29 05:07 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (746 votes), past polls