Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^4: Tie::File failing with Unicode/UTF-8 encoding?

by remiah (Hermit)
on Nov 05, 2012 at 22:10 UTC ( #1002394=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Tie::File failing with Unicode/UTF-8 encoding?
in thread Tie::File failing with Unicode/UTF-8 encoding?

Hello, HellenCr.

I have read Unk's post at stackoevrflow. This is not trivial problem for me having been troubled and struggled with encoding/decoding issue for a long time. It seems Glu monks not noticing this thread...

Unk says

1. Encode manually before handing off to the tied array
2. Figure out what the issue is with Tie::File
No 1 must be like mine, wrapping Tie::File with accessor methods. For No.2, I wish some superior monks pursuit whether it is really seek problem, as Unk says. And there could be No.3 using DB_File module.
#!/usr/bin/perl # DB_File example from http://blog.livedoor.jp/dankogai/archives/51500 +530.html use 5.010; use strict; use warnings; use utf8; use Fcntl; use DB_File; use DBM_Filter; my @Tied; my $Filename='087.txt'; my $db = tie @Tied, 'DB_File', $Filename, O_CREAT | O_RDWR, 0644, $DB_ +RECNO; $db->Filter_Push('encode' => 'UTF-8'); binmode STDOUT,':encoding(UTF-8)'; my $i =0; while (<DATA>) { chomp; $Tied[$i] = $_; ++$i; } # end while (<DATA>) $i =0; foreach (@Tied) { say "$i $Tied[$i]" if /&#964;/; #greek letter... ++$i; } # end foreach (@Tied) $db->Filter_Pop(); untie $Filename; __DATA__ your greek input
Referenced page written in Japanese. Filter_Push seems working fine for me... but it is really greek for me.

regards.


Comment on Re^4: Tie::File failing with Unicode/UTF-8 encoding?
Download Code
Re^5: Tie::File failing with Unicode/UTF-8 encoding?
by HelenCr (Monk) on Nov 07, 2012 at 10:29 UTC

    Remiah: thank you for your latest post - it's great, I learned a lot from it.

    But, there is still a little problem. Consider the following script:

    use strict; use warnings; use 5.014; use Win32::Console; use autodie; use warnings qw< FATAL utf8 >; use Carp; use Carp::Always; use utf8; use Fcntl; use DB_File; use DBM_Filter; my ($i, $FileName); my (@Tied); binmode STDOUT, ':encoding(UTF-8)'; binmode STDERR, ':encoding(UTF-8)'; binmode $DB::OUT, ':encoding(UTF-8)' if $DB::OUT; # for the debugger Win32::Console::OutputCP(65001); # Set the console code page t +o UTF8 $FileName = 'E:\\My Documents\\Technical\\Perl\\Eclipse workspace\\Wor +k\\'Tie File test.txt'; my $db = tie @Tied, 'DB_File', $FileName, O_CREAT | O_RDWR, 0644, $DB_ +RECNO; $db->Filter_Push('encode' => 'UTF-8'); while (<DATA>) { push @Tied, $_; } # end while (<DATA>) $i =0; foreach (@Tied) { print "$i $_"; ++$i; } # end foreach (@Tied) $db->Filter_Pop(); untie $FileName; __DATA__ &#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;; &#960;&#940;&#961;&#964;&#949; &#964;&#959; &#942; &#945;&#966;&#942;& +#963;&#964;&#949; &#964;&#959; &#1513;&#1500;&#1493;&#1501; &#1495;&#1489;&#1512;&#1497;&#1501; abc &#1500;&#1488; &#1499;&#1503;&#1499;&#1503; efg &#1502;&#1514;&#1497; &#1493;&#1500;&#1488;&#1503; This is it &#1502;&#1506;&#1499;&#1513;&#1497;&#1493; &#1500;&#1506;&#1499;&#1513 +;&#1497;&#1493; &#931;&#942;&#956;&#949;&#961;&#945; &#949;&#943;&#957;&#945;&#953; &# +932;&#961;&#943;&#964;&#951; &#920;&#941;&#955;&#969; &#957;&#945; &#966;&#940;&#969; &#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;;

    (It's practically identical to your latest script). Notice that on purpose, I took away the "chomp" on input to the tied array, and indeed, as you can see from the script print (STDOUT) output, the array members do have "\r\n" terminations.

    But, inspecting the saved tied file, it turns out that the records _do_not_ have newlines (or \r\n) terminations!

    Why is that, and what should be done? (Maybe something in $db->Filter_Push())? And how?

    In addition, can you refer us to a good documentation for DB_File and DBM_Filter modules?

    Many TIA

    Helen

      Hello HelenCR.

      I just thought setting bval option would solve this crlf problem, but DB_File allows only one byte for record delimitor (sorry, I didn't know this ...). DB_File document says

      Also note that the bval option only allows you to specify a single byte as a delimiter.

      So as you say, Filter_Push for crlf will work.

      my @Tied; my $Filename='087.txt'; my $dbh = new DB_File::RECNOINFO ; $dbh->{bval}="\n"; #set bval explicitly my $db = tie @Tied, 'DB_File', $Filename, O_CREAT | O_RDWR, 0644, $dbh or die "failed top open $Filename"; #utf8 $db->Filter_Push('encode' => 'UTF-8'); #add crlf filter $db->Filter_Push( Fetch => sub { s/\r\n$/\n/;}, #convert crlf to lf Store => sub { $_ .="\r"; }, #add cr ); # ... same as before
      For documents, I like HTML pages of search.cpan.org
      DB_File
      DBM_Filter
      And you will have "t" directory for each modules, which is for testing. Scripts in "t" directory gives me hints for usages.

      regards, good luck.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1002394]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2014-09-15 03:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (145 votes), past polls