Re^5: Tie::File failing with Unicode/UTF-8 encoding?

[HelenCr]

Remiah: thank you for your latest post - it's great, I learned a lot from it.

But, there is still a little problem. Consider the following script:

use strict;  use warnings;
 use 5.014;    
 use Win32::Console;
 use autodie; 
 use warnings qw< FATAL utf8 >;
 use Carp;  use Carp::Always;
 use utf8;
use Fcntl;
use DB_File;
use DBM_Filter;

my ($i, $FileName);
my (@Tied);
binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';
binmode $DB::OUT, ':encoding(UTF-8)' if $DB::OUT; # for the debugger
Win32::Console::OutputCP(65001);         # Set the console code page t
+o UTF8

$FileName = 'E:\\My Documents\\Technical\\Perl\\Eclipse workspace\\Wor
+k\\'Tie File test.txt';
my $db = tie @Tied, 'DB_File', $FileName, O_CREAT | O_RDWR, 0644, $DB_
+RECNO;
$db->Filter_Push('encode' => 'UTF-8');

while (<DATA>) {
    push @Tied, $_;
} # end while (<DATA>) 

$i =0;
foreach (@Tied) {
    print "$i $_";
    ++$i;
} # end foreach (@Tied)
$db->Filter_Pop();
untie $FileName;
__DATA__
&#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;;
&#960;&#940;&#961;&#964;&#949; &#964;&#959; &#942; &#945;&#966;&#942;&
+#963;&#964;&#949; &#964;&#959;
&#1513;&#1500;&#1493;&#1501; &#1495;&#1489;&#1512;&#1497;&#1501;
abc &#1500;&#1488; &#1499;&#1503;&#1499;&#1503; efg
&#1502;&#1514;&#1497; &#1493;&#1500;&#1488;&#1503; This is it
&#1502;&#1506;&#1499;&#1513;&#1497;&#1493; &#1500;&#1506;&#1499;&#1513
+;&#1497;&#1493; 
&#931;&#942;&#956;&#949;&#961;&#945; &#949;&#943;&#957;&#945;&#953; &#
+932;&#961;&#943;&#964;&#951;
&#920;&#941;&#955;&#969; &#957;&#945; &#966;&#940;&#969;
&#964;&#953; &#954;&#940;&#957;&#949;&#964;&#949;;

(It's practically identical to your latest script). Notice that on purpose, I took away the "chomp" on input to the tied array, and indeed, as you can see from the script print (STDOUT) output, the array members do have "\r\n" terminations.

But, inspecting the saved tied file, it turns out that the records _do_not_ have newlines (or \r\n) terminations!

Why is that, and what should be done? (Maybe something in $db->Filter_Push())? And how?

In addition, can you refer us to a good documentation for DB_File and DBM_Filter modules?

Many TIA

Helen

[remiah]

Hello HelenCR.

I just thought setting bval option would solve this crlf problem, but DB_File allows only one byte for record delimitor (sorry, I didn't know this ...). DB_File document says


Also note that the bval option only allows you to specify a single byte as a delimiter.

So as you say, Filter_Push for crlf will work.


my @Tied;
my $Filename='087.txt';

my $dbh = new DB_File::RECNOINFO ;
$dbh->{bval}="\n";                  #set bval explicitly
my $db = tie @Tied, 'DB_File', $Filename, O_CREAT | O_RDWR, 0644, $dbh
    or die "failed top open $Filename";

#utf8
$db->Filter_Push('encode' => 'UTF-8');

#add crlf filter
$db->Filter_Push(
    Fetch => sub { s/\r\n$/\n/;}, #convert crlf to lf
    Store => sub { $_ .="\r"; },  #add cr
);
# ... same as before

For documents, I like HTML pages of search.cpan.org
DB_File
DBM_Filter
And you will have "t" directory for each modules, which is for testing. Scripts in "t" directory gives me hints for usages.

regards, good luck.