Encoding question

Hello, I am trying to work with a XML document and having some encoding-type issues with it. First, this is the first few bytes of the XML document passed through "od -c":


0000000 377 376   <  \0   ?  \0   x  \0   m  \0   l  \0      \0   v  \
+0
0000020   e  \0   r  \0   s  \0   i  \0   o  \0   n  \0   =  \0   "  \
+0
0000040   1  \0   .  \0   0  \0   "  \0      \0   e  \0   n  \0   c  \
+0
0000060   o  \0   d  \0   i  \0   n  \0   g  \0   =  \0   "  \0   u  \
+0
0000100   t  \0   f  \0   -  \0   1  \0   6  \0   "  \0   ?  \0   >  \
+0
0000120  \r  \0  \n  \0   <  \0   l  \0   i  \0   s  \0   t  \0   i  \
+0
0000140   n  \0   g  \0   _  \0   f  \0   e  \0   e  \0   d  \0   >  \
+0
0000160  \r  \0  \n


[download]

Of relevance here, I suppose, is that it is encoding="utf-16", which appears to me the XML document is correctly formed, although I am certainly no expert in this area. In plain ASCII, this would look something like:

<?xml version="1.0" encoding="utf-16"?>
<listing_feed>
[download]

Now, I'm trying to do a regular expression match on this string, like so:

if($string =~ m/xml/)
{
  ...
}
[download]

.. but this does not work... However, this does:

if($string =~ m/x.m.l./)
{
   ...
}
[download]

Have I completely misunderstood that this type of regular expression match should work? I've tried using the "Encode" module to re-code this to UTF-8, or even ASCII but nothing I do can make this work.

.. I think I must be missing something obvious here...

- dEvNuL

Comment on Encoding question Select or Download Code
Re: Encoding question by ikegami (Pope) on Nov 07, 2006 at 01:08 UTC
Perl has two types of strings. Strings of bytes and strings of characters. You wish to match characters, but you're matching bytes. Decode (using Encode's `decode` function, for example) the bytes into characters. `use Encode qw( decode ); my $bytes = "x\0m\0l\0"; print(length($bytes), "\n"); # 6 my $chars = decode('utf16le', $bytes); print(length($chars), "\n"); # 3 print($chars =~ /xml/ ?1:0,"\n"); # 1` [download] Update: Oops, I had `utf8` instead of `utf16le` originally. `utf16` will also work with the original string since it contained a BOM.	[reply] [d/l] [select]
Re: Encoding question by Errto (Vicar) on Nov 07, 2006 at 01:22 UTC
Since it seems like this is coming from a file, the fact that it shows up that way in Perl suggests that you're not reading the file in its actual encoding. To do that, simply open the file like so: `open my $fh, '<:encoding(utf16le)', $filename;` [download] Also, since your file seems to have a BOM at the beginning, you may need to do a little `s/^\x{feff}//` [download] on the first line of the file.	[reply] [d/l] [select]
Re: Encoding question by graff (Canon) on Nov 07, 2006 at 02:42 UTC
To supplement the good advice given above... I've tried using the "Encode" module to re-code this to UTF-8, or even ASCII but nothing I do can make this work. The previous replies have given enough info to read the file correctly. If you want to write the data to a new file using utf8 instead of utf-16le, you'd want to make sure to set the output file handle to utf8 mode, and replace the "utf-16" in the opening xml tag with "utf-8": `open(IN,"<:encoding(utf-16le)","your_file.xml"); { local $/; $_ = <IN>; } s/(xml version="1.0" encoding="utf)-16/$1-8/; # (that was cheating, but what the heck) # do whatever else needs to be done with the data # -- but use a real xml parser for that... then: binmode STDOUT, ":utf8"; # or whatever file handle you need to use print;` [download]	[reply] [d/l]


Welcome to the Monastery
	PerlMonks

Encoding question

The best material for plates (tableware) is: