Re^6: encoding problem om Ubuntu Linux

[j.goor]

I't getting more and more complex... (putty was somehow reset to something other than UTF8 which made it all more tricky

1) the directory itself
root@slarti:/media/usbdisk/music_org/scripts/test# ls /media/usbdisk/music/checkit/Crosby\,\ Stills\,\ Nash\ \&\ Young/
Déjà Vu

So... that seems to be Ok (assuming this correctly being viewd while at UTF8)

2) The source file's content (part of it at least)
root@slarti:/media/usbdisk/music_org/scripts/test# grep 'Crosby\,\ Stills\,\ Nash\ \&\ Young' mp3_v4.csv
/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?▒ Vu;Carry On;1;Rock;1970

Now, still being in UTF8 mode this does not look good

3) running a check-script of mine...
#!/usr/bin/perl

# INIT ----
use strict;
use warnings;

# VARS ----

# SUBS ----


# MAIN ----
print "Opening list...\n";
my $file = shift;

open (F, $file) or die "$0\n";
while (my @fields = split /;/, <F>, 8) {
unless (-f $fields[0]) {
print $fields[0] . ": not found\n";
}
}
close (F);

print "Ready\n";


Output: (part of it)
root@slarti:/media/usbdisk/music_org/scripts/test# ./check_paths.pl mp3_v4.csv | grep Stills /media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/D▒j▒ Vu/01 - Carry On.mp3: not found

This means the path in the sourcefile does not match the actual subdirectory

4) The fun part: vi mp3_v2.csv
/media/usbdisk/music/checkit/Crosby, Stills, Nash & Young/Déjà Vu/01 - Carry On.mp3;Crosby, Stills, Nash & Young;D?j?ï Vu;Carry On;1;Rock;1970

Compare this output to step 2 which does a grep on this very same file. Now all of a sudden it's looking good again. The source files has not been edited and all these outputs have been taken in the same putty windows while check before and aft on the encoding. UTF8 all the time.

Now I'm puzzled.
I think I'm gonna rub Buddha's belly - hopefully it'll provide me with some luck :-)

[almut]

Rather than trying to come up with explanations for why get what you get...  let me just recommend that, when debugging encoding problems, you start by looking at the raw data¹ with as few tools in between as possible. Every tool (terminal, putty, grep, vi, browser, whatever) or processing step might have it's own problems or automagic-isms with one or the other encoding, so what you see may not necessarily be what you really have.

My preferred tool in cases like these is a classical hexdump, or as ikegami suggested, a Devel::Peek dump (when in Perl). Together with some knowledge of how the various encodings represent data, this usually allows you to figure out what's going wrong, eventually.

In this particular case, I would start by looking at a hexdump of the CSV file (e.g. with the command line tool hexdump, which is installed/available on most distros). Then we would reliably know what the "Déjà" is represented as in the CSV file.

___

¹ though Juerd might disagree, saying that you shouldn't, unless you're an expert already...

[moritz]

What are the values you get when typing

:set encoding
:set fileencoding

in vim?

What do you get when you simply enter locale on the command line?

[j.goor]

root@slarti:/media/usbdisk/music_org/scripts/test# localev LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=


:set encoding
encoding=utf-8

:set fileencoding
fileencoding=latin1

[moritz]

Well, if vim says the file is encoded in latin1, I believe that. Try to use Encode to or open my $handle, '<:encoding(ISO-8859-1)', $file to decode it, and then re-encode it as utf-8. It also explains why cat'ting the file doesn't display the file names correctly.

See my reply on your original question for more details on de- and encoding.