Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^2: Tie::File failing with Unicode/UTF-8 encoding?

by remiah (Hermit)
on Nov 04, 2012 at 02:24 UTC ( #1002156=note: print w/ replies, xml ) Need Help??


in reply to Re: Tie::File failing with Unicode/UTF-8 encoding?
in thread Tie::File failing with Unicode/UTF-8 encoding?

Hello.

I thought getter, setter method will avoid this trouble. Below was my trial. MyTieFile.pm

package MyTieFile; use strict; use warnings; use Tie::File; use Encode qw/encode decode/; use Carp; my @Tied; sub new { my $class=shift; my $filename=shift; my %args =@_; my %self; $self{filename}=$filename; if( exists($args{discipline}) ){ $self{encode}=$args{discipline}; if( $self{encode} =~ /\((.*)\)/){ $self{encode}=$1; } delete $args{discipline}; } tie(@Tied, 'Tie::File', $filename, %args) or confess 'tie @Tied failed'; return bless(\%self, $class); } sub get{ my $self=shift; my $idx=shift; return $self->{encode} ? decode($self->{encode}, $Tied[$idx]) : $Tied[$idx]; } sub set{ my $self=shift; my $idx=shift; my $val=shift; $Tied[$idx]= $self->{encode} ? encode($self->{encode}, $val) : $val; } sub max_index{ my $self=shift; return $#Tied; } sub DESTROY { my $self=shift; untie $self->{filename}; } 1;
And test script
use strict; use warnings; use utf8; use 5.012; use feature qw< unicode_strings>; use MyTieFile; binmode STDOUT, ':unix:utf8'; my $t=MyTieFile->new('087.txt',recsep => "\x0D\x0A", discipline => ':e +ncoding(UTF-8)'); my $i; $i =0; while (<DATA>) { chomp; $t->set( $i, $_); ++$i; } # end while (<DATA>) $i =0; foreach ( 0 .. $t->max_index) { say "$i ".$t->get($i); ++$i; } # end foreach (@Tied) __DATA__ .. pasted UTF8 text from stackoverflow ... I wish perlmonk support UTF +-8 characters.

And I saw your post. So, overriding STORE,FETCH seems more elegant. I tried your untested code but it seems producing same warnings message for utf8 characters.

Why???

This seems to be a known problem because there is stackoverflow thread and I see the name "ikegami"

I hope "ikegami" or someone explains this problem , a little more...


Comment on Re^2: Tie::File failing with Unicode/UTF-8 encoding?
Select or Download Code
Re^3: Tie::File failing with Unicode/UTF-8 encoding?
by Anonymous Monk on Nov 04, 2012 at 12:55 UTC

    And I saw your post. So, overriding STORE,FETCH seems more elegant. I tried your untested code but it seems producing same warnings message for utf8 characters.

    Why???

    I don't know , I didn't test it :)

      Most probably, this is the warning (or similar):

      utf8 "\xCE" does not map to Unicode at F:/Win7programs/Dwimperl/perl/l +ib/Tie/File.pm line 917 Tie::File::_read_record('Tie::File=HASH(0x24cb72c)') called at + F:/Win7programs/Dwimper l/perl/lib/Tie/File.pm line 175 Tie::File::_fetch('Tie::File=HASH(0x24cb72c)', 0) called at F: +/Win7programs/Dwimperl/p erl/lib/Tie/File.pm line 210 Tie::File::STORE('Tie::File=HASH(0x24cb72c)', 0, '&#964;&#953; + &#954;&#940;&#957;&#949;&#964;&#949;;') called at tie file test .pl line 31

      Can you look into it and try to suggest the cause?

        Most probably, this is the warning (or similar): Can you look into it and try to suggest the cause?

        Sorry, no, its too clever for me. I posted what I've been able to make work, for the rest complain to maintainers.

Re^3: Tie::File failing with Unicode/UTF-8 encoding?
by HelenCr (Monk) on Nov 05, 2012 at 17:20 UTC

    remiah: I studied your module and test script: you've done a very good job - it's working. Thank you for that.

    But, what this effectively does (as UNK noted in his answer here: http://stackoverflow.com/questions/13209474/ ), is re-encoding the data before inserting it into the tied array and the tied file; so the array does not contain Unicode data in internal Perl representation, but instead simply contains the imported UTF-8 strings.

    Now in my project, I am doing regex comparisons and substitutions against the tied array; so if I go this route, I'll have to re-decode the array element before any processing, and re-encode it again.

    What do you think?

    Many thanks for your well-thought-out answer.

    Helen

      Hello, HellenCr.

      I have read Unk's post at stackoevrflow. This is not trivial problem for me having been troubled and struggled with encoding/decoding issue for a long time. It seems Glu monks not noticing this thread...

      Unk says

      1. Encode manually before handing off to the tied array
      2. Figure out what the issue is with Tie::File
      
      No 1 must be like mine, wrapping Tie::File with accessor methods. For No.2, I wish some superior monks pursuit whether it is really seek problem, as Unk says. And there could be No.3 using DB_File module.
      #!/usr/bin/perl # DB_File example from http://blog.livedoor.jp/dankogai/archives/51500 +530.html use 5.010; use strict; use warnings; use utf8; use Fcntl; use DB_File; use DBM_Filter; my @Tied; my $Filename='087.txt'; my $db = tie @Tied, 'DB_File', $Filename, O_CREAT | O_RDWR, 0644, $DB_ +RECNO; $db->Filter_Push('encode' => 'UTF-8'); binmode STDOUT,':encoding(UTF-8)'; my $i =0; while (<DATA>) { chomp; $Tied[$i] = $_; ++$i; } # end while (<DATA>) $i =0; foreach (@Tied) { say "$i $Tied[$i]" if /&#964;/; #greek letter... ++$i; } # end foreach (@Tied) $db->Filter_Pop(); untie $Filename; __DATA__ your greek input
      Referenced page written in Japanese. Filter_Push seems working fine for me... but it is really greek for me.

      regards.

        Remiah: thank you for your latest post - it's great, I learned a lot from it.

        But, there is still a little problem. Consider the following script:

        use strict; use warnings; use 5.014; use Win32::Console; use autodie; use warnings qw< FATAL utf8 >; use Carp; use Carp::Always; use utf8; use Fcntl; use DB_File; use DBM_Filter; my ($i, $FileName); my (@Tied); binmode STDOUT, ':encoding(UTF-8)'; binmode STDERR, ':encoding(UTF-8)'; binmode $DB::OUT, ':encoding(UTF-8)' if $DB::OUT; # for the debugger Win32::Console::OutputCP(65001); # Set the console code page t +o UTF8 $FileName = 'E:\\My Documents\\Technical\\Perl\\Eclipse workspace\\Wor +k\\'Tie File test.txt'; my $db = tie @Tied, 'DB_File', $FileName, O_CREAT | O_RDWR, 0644, $DB_ +RECNO; $db->Filter_Push('encode' => 'UTF-8'); while (<DATA>) { push @Tied, $_; } # end while (<DATA>) $i =0; foreach (@Tied) { print "$i $_"; ++$i; } # end foreach (@Tied) $db->Filter_Pop(); untie $FileName; __DATA__ &#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;; &#960;&#940;&#961;&#964;&#949; &#964;&#959; &#942; &#945;&#966;&#942;& +#963;&#964;&#949; &#964;&#959; &#1513;&#1500;&#1493;&#1501; &#1495;&#1489;&#1512;&#1497;&#1501; abc &#1500;&#1488; &#1499;&#1503;&#1499;&#1503; efg &#1502;&#1514;&#1497; &#1493;&#1500;&#1488;&#1503; This is it &#1502;&#1506;&#1499;&#1513;&#1497;&#1493; &#1500;&#1506;&#1499;&#1513 +;&#1497;&#1493; &#931;&#942;&#956;&#949;&#961;&#945; &#949;&#943;&#957;&#945;&#953; &# +932;&#961;&#943;&#964;&#951; &#920;&#941;&#955;&#969; &#957;&#945; &#966;&#940;&#969; &#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;;

        (It's practically identical to your latest script). Notice that on purpose, I took away the "chomp" on input to the tied array, and indeed, as you can see from the script print (STDOUT) output, the array members do have "\r\n" terminations.

        But, inspecting the saved tied file, it turns out that the records _do_not_ have newlines (or \r\n) terminations!

        Why is that, and what should be done? (Maybe something in $db->Filter_Push())? And how?

        In addition, can you refer us to a good documentation for DB_File and DBM_Filter modules?

        Many TIA

        Helen

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1002156]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (10)
As of 2014-07-28 08:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (193 votes), past polls