character encoding & french accents

[dstamos]

Hello, I hope that someone can help me with this. I am not a perl programmer but rather the owner of a purchased perl (cgi) script.

I am working with the programmer on this but we are having a hard time figuring out where the problem is exactly (local, cgi script or smtp).

The application is mailing list script that connects to a smtp server to send the email. The emails are written in french with accents. When we receive the email it is garbled in our email program and webmail. Changing character encoding in our browser or gui to UTF-8 makes the accents readable. The standard in browsers and gui's would seem to be iso-8559-1.

This script writes the email to a text file first and then sends it. Its running on a Fedora Core 3 server with Apache 2.0. Is the problem with the way the file is encoded ? or is it a local problem on the server ?

Something i noticed is that when i vi the text file on the server the french accents are garbled.

does anyone have experience with this ?

regards

Denny

[leocharre]

dont ever again ever ever pay money for software

pay for someone to code for you custom- or to help you with code. but if you pay for a piece of software.. SUCKER- serves you right. hahahahaha.

look for an alternative solution to your problem- if yours was an open source script then it would probably not be all f&&ed up like that to begin with, and if it were, it would get fixed by sundown at no cost to your silly &ss- i mean.. it's one thing if you mother buys a copy of windows xp or her latest 'make your own barby' program.. but here ?

This is more the uhmm.. open source community. I think you should start by telling us what your problem is .. then someone can suggest where to look. Sorry you made a mistake paying for software. Don't do it again.

you come to perl, perl come to yoo- you pay mulah for software perl no likey, but we help yoo..

just STOP paying money for sofware.

Considered: astaines: Personally abusive and rather silly - fit for reaping or editing?
Unconsidered: g0n - enough keep votes (7/0/3)

[dstamos]

leocharre,

Is this forum for helping people or bashing them ??!

FYI the script is not all f&&ed up as you say, you are just wasting my time with your nonsense.

[g0n]

leocharre,

there is a case to be made for the advantages of open source over commercially developed software. But you aren't making it, and you certainly aren't helping it by being borderline abusive and deliberately unhelpful. You can find a pretty good summary in the Richard Stallman quote from DigitalKitty's homenode:

"Proprietary software is unethical, because it denies the user the basic freedom to control his/her own computer and to cooperate. The problem with proprietary software is that a specific developer controls its development--you, the user, do not."
-- Richard Stallman


There are many reasons why someone would show up here looking for help with a piece of code they've paid for - perhaps the OP is looking at a piece of code that a freelancer has written? Or perhaps (he|she) or (his|her) boss has made a (possibly ill advised) choice to buy a piece of code that provides a particular solution.

I'm as much an open source zealot as the next monk, but perlmonks is a site about perl, not necessarily open source. Please bear that in mind, and either try to be polite and helpful to people asking for help, or if you can't, hold your tongue.

[leocharre]

I just feel that the open source community shoots itself in the foot by doing things like porting 'free' apps to proprietary operating systems- and fixing broken proprietary suff. I kiss Richard Stallman a55. I just don't have his, or your- dipplomacy and tact. What I wrote seems to have gotten -26 votes. This *is* a community, not a graffitti wall. So I will take note and learn from it. Thank you, g0n.

[kutsu]

You'll need to set the charset as unicode (you can do this with a meta tag in the head <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">) btw. you can find more info at unicode.org (check out the FAQs).

[rhesa]

I have plenty experience with mismatching character encodings, and I agree that it can be hard to track down.

First of all, where does your source document come from? Which factors determine its encoding?

Ultimately the best course of action is to ensure that you're using utf8 everywhere (and advertising that to the viewing programs). You can tell the web browser that your content is in utf8 by sending an additional "charset" header attribute like this:

  use CGI;
  my $cgi = CGI->new;

  ...

  print $cgi->header(-type=>'text/html', -charset=>'utf-8');

You can do the same for emails, by giving them the appropriate MIME headers:

Content-Type: text/plain; charset="utf-8"

How you do that depends on how your email-sending code is designed.

I'd like to point out that in recent perls (version 5.8), under most circumstances, strings are encoded internally as utf8. So it makes sense to be consistent about that in the rest of your application.

[dstamos]

Thank you to rhesa, graff & fraktalisman for your constructive input. "First of all, where does your source document come from? Which factors determine its encoding?" The content comes from an input box (form) and the cgi program writes this to a text file. What happens after that i dont know. I was able to modify the script to insert a MIME header as you suggested but it did not change anything. I dont know what determines its encoding. I think im going to have to get the programmer involved here and come back with some code and more info. thank you again to everyone.

[rhesa]

I think getting your developer in here to discuss the details is a very good idea.

I've found on several occasions that it's necessary to manually upgrade form input to utf8. For some reason, CGI returns raw byte strings, and those might end up being upgraded to utf once more, resulting in lots of squiggly characters. I did this like so:

    use Encode;
    my $email_body = $cgi->param( 'email_body' );
    $email_body    = decode_utf8( $email_body );

After this, most conversions and display issues are a snap.

[graff]

thesa's advice is basically correct, but your point iso-8859-1 being the "default" encoding for many apps is also relevant. It's true that more and more apps (esp. browsers and browser-based email clients) are making it easier to change character encoding at the display as needed, but it's also true that a lot of them (and the people who use them) still treat the legacy iso-8859 and cp12.. as the "default".

If you would prefer your email output to be iso-8859-1, use the Encode module on the text data, or PerlIO control on the output file handle, in order to assure that the text is written with that encoding; assuming the email text is going out via a file handle the easiest thing would probably be:

binmode $fh, ":encoding(iso-8859-1)";  # convert internal utf8 to lati
+n1 on output

If you do this, you should still heed thesa's advice, and explicitly declare what character set you're using in the MIME header.

[fraktalisman]

I've had similar problems recently when upgrading an existing content management system from Latin-1 to UTF-8. After extensive testing we have decided to display all website content in UTF-8 encoding, but send emails in Latin-1, because many freemail websites seem to ignore the charset directives.

If your provider happens to run an older Perl version, the Encode module might not support UTF-8 or it might not be there at all. In that case, have a look at the sub latin1 in the following example: Converting character encodings.