problem decoding a quoted-printable euc-jp mail header

[blahblahblah]

Hi,

I understand a bit about how encoded email data works, but I'm far from an expert. I'm stumped on a problem that just started occurring for one of our customers. My script isn't able to parse quoted-printable, euc-jp mail subjects. It does fine with other asian charsets + quoted-printable, and also with euc-jp base64 data.

The data that I have is a little fishy looking to me, and I'm wondering if someone can offer any insight into what's different between my working and non-working strings below, and/or advice on how to decode the non-working string.

use strict;
use warnings;
use Encode;

for my $subject (
# bad:
              '=?euc-jp?Q?=83v=83=8D=83t=83B=81=5B=83=8B?=',
# bad:
              '=?euc-jp?Q?=97L=97=BF=83T=81=5B=83r=83X?=',
# good:
                 '=?euc-jp?Q?=B2=F1=B0=F7=CD=CD=A4=D8=C9=AC=A4=BA=C5=F
+6=A4=EB?=',
                )
{
    print "\ndecoding $subject ...\n";
    my $decoded = Encode::decode('MIME-Header',$subject);
    print "result:    $decoded\n";
}
__END__

decoding =?euc-jp?Q?=83v=83=8D=83t=83B=81=5B=83=8B?= ...
result:    \x83v\x83\x8D\x83t\x83B\x81[\x83\x8B

decoding =?euc-jp?Q?=97L=97=BF=83T=81=5B=83r=83X?= ...
result:    \x97L\x97\xBF\x83T\x81[\x83r\x83X

decoding =?euc-jp?Q?=B2=F1=B0=F7=CD=CD=A4=D8=C9=AC=A4=BA=C5=F6=A4=EB?=
+ ...
Wide character in print at ./test line 15.
result:  [something that looks like "correct" garbled text in my shell
+]

The thing that looks strange about the bad strings is the alternation of "=xx" with single ascii characters. I think technically that's valid quoted-printable, but usually I just see the repeated "equals sign plus 2 hex chars" like in the good string above.

Also, when I bounce this mail to my outlook, it shows the subject correctly.

Any advice?

Thanks,
Joe

[graff]

I think the problem may be that you are actually dealing with two different encodings for Japanese. If the first two examples in your list were identified as "shiftjis" instead of "euc-jp", they would produce valid Japanese characters rather than gibberish. (Whether they produce coherent, meaningful Japanese subject lines is another matter, and I'm sorry I can't help you there.)

As for the third example, euc-jp works correctly for that, and changing its label to shiftjis causes it to come out as gibberish.

If I understand the docs I've seen, shiftjis is a system that uses one byte per character for ASCII content, and two bytes per character for Japanese, where the first byte of a Japanese pair always has the eigth bit set, and the second byte may or may not have the eigth bit set.

Meanwhile, euc-jp uses a different strategy to allow ASCII and Japanese content to coexist: ASCII content is again one byte per character, but Japanese content may be two or three bytes per character, and all bytes of a Japanese character must have the eigth bit set.

Since your first two strings appear to be composed of byte pairs with the second byte sometimes in the ASCII range, they cannot be valid euc-jp. They'll display nicely if treated as shiftjis (and then if they make sense to people who know Japanese, so much the better).

As for your finding that they all display correctly in Outlook... euc-jp comes from the unix world, whereas shiftjis is more in the MS-Windows domain. I guess I wouldn't be surprised to learn that MS software somehow manages to ignore an incorrect encoding label.

So where are those labels coming from, anyway? Is someone sending you bad data, or are you creating bad data?

[blahblahblah]

Thanks, your suggestion worked perfectly. If I add this, the data comes out correctly:

                $str =~ s{euc-jp}{cp932};

    So where are those labels coming from, anyway?

These mails are coming from hotmail. Just last week they started coming in as 'euc-jp?Q?' instead of 'euc-jp?B?', and the base64 ones were working fine. I don't know if hotmail changed something or if some email server or spam filter at my client's site might be altering the message. I've seen similar problems with the bodies of some mails from yahoo and hotmail before, where the charset is labeled as iso-8859-1 but it's really some Japanese or Korean encoding. Whatever else people might say about MS, Outlook does a great job of 'dwim' when it comes to email, for better or worse.

[Melly]

I know nothing of which you speak, but I noticed in the Mime-Header documentation:

"It would be nice to support encoding to non-UTF8, such as =?ISO-2022-JP? and =?ISO-8859-1?= but that makes the implementation too complicated. These days major mail agents all support =?UTF-8? so I think it is just good enough."

Any help?