Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: How to concatenate utf8 safely?

by gregor42 (Parson)
on Oct 25, 2016 at 13:22 UTC ( #1174690=note: print w/replies, xml ) Need Help??


in reply to How to concatenate utf8 safely?

andal : " ... First of all, you have to worry about representation of characters in the octets that you receive from external applications. That depends on locale settings ... "

OP " ... I assume that the problem is my code and not the data coming in since one can usually depend on people to get their own names right ... "

It would appear that my initial assumption was incorrect. I challenged that & as it turns out what I am dealing with is a mixture of localized character sets taken as input from across Europe, cut & pasted between spreadsheets in an HR department spanning multiple offices.

(ノωノ)

These are conscientious people, mind you, who are concerned about getting the characters just right by potentially editing with multiple programs along the way...

I'm glad that I asked and should have done so sooner.

Now, in the proper mindset and having done my revision a big thing I was missing was that I was using :utf8 instead of :encoding(utf8) which allowed me to regain the trust factor in the data.

I had all kinds of stupid ideas and bad assumptions that led me to chase phantoms. Now at least I can identify mangled input on the way in.



Wait! This isn't a Parachute, this is a Backpack!

Replies are listed 'Best First'.
Re^2: How to concatenate utf8 safely?
by choroba (Archbishop) on Oct 25, 2016 at 21:15 UTC
    > :utf8 instead of :encoding(utf8)

    Switch to :encoding(UTF-8) , it's even safer. See Re^2: Read and write UTF-8 for an example.

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1174690]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (3)
As of 2020-06-05 19:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (40 votes). Check out past polls.

    Notices?