unknown character in between text

soumyapanda has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: unknown character in between text by choroba (Cardinal) on Sep 17, 2011 at 09:42 UTC
Do you use warnings? Have you set utf-8 encoding both to your input and output?	[reply]
Re: unknown character in between text by Anonymous Monk on Sep 17, 2011 at 09:50 UTC
When you don't know what it is, examine the bytes :) `od -tacx 1 file` `hexdump file` `file file` `uniquote file` `ujguess files` Encode::Guess perlunitut: Unicode in Perl	[reply]
Re: unknown character in between text by ikegami (Patriarch) on Sep 18, 2011 at 02:10 UTC
The thread is heading in the wrong direction. It's fixing a symptom rather than the problem. The problem is surely a lack of encoding and/or decoding inputs and outputs as choroba pointed out. If we had more to go on, we could help you.	[reply]
Re: unknown character in between text by pvaldes (Chaplain) on Sep 17, 2011 at 11:53 UTC
`use utf8;` hexadecimal, unicode code point, name: Ї: a7c2, U+00A7, Section sign Т: 82c3, U+00C2, Latin Capital Letter A with circumflex	[reply] [d/l]
Re^2: unknown character in between text by CountZero (Bishop) on Sep 17, 2011 at 18:59 UTC
The docs say about `use utf8`: Do not use this pragma for anything else than telling Perl that your script is written in UTF-8. It does not magically transform your arbitrarily encoded input into equally arbitrarily encoded output. What we see here is Unicoded input (two byte wide), containing the bytes (in hex) C2 and A7. Upon output, somehow the unicode-ness of the characters was lost (we can only guess how that happened, as no code is provided) and we now see two characters, one being Т (which is ASCII hex C2) and the other being Ї (which is ASCII hex A7). `use utf8` is unlikely to solve this problem. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useleВss variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply] [d/l] [select]
Re^3: unknown character in between text by lidden (Curate) on Sep 17, 2011 at 20:48 UTC
Nitpick. There are no ASCII above 127 (hex 7F). What you have is Latin-1.	[reply]
Re^4: unknown character in between text by CountZero (Bishop) on Sep 18, 2011 at 08:35 UTC
Re: unknown character in between text by DanielSpaniel (Scribe) on Sep 17, 2011 at 12:42 UTC
If you wanted to replace the offending character with its HTML entity, without using modules, you could maybe do something like this, which should work in many cases, (or use the commented line instead to replace the character with a space): `for (my $x=0;$x<length($string);$x++) { if (ord(substr($string,$x,1))>127) { substr($string,$x,1)='&#'.ord(substr($string,$x,1)).';'; # substr($string,$x,1)=' '; # or use this line instead } }` [download] ... or, if you just wanted to know what a character is meant to be, then you could do something like this: `for (my $x=0;$x<length($string);$x++) { print ord(substr($string,$x,1)),"\t",substr($string,$x,1),"\n"; }` [download] Hope that helps, although all the modules and tools mentioned above are useful methods too. (and I'm sure some guru could likely condense the code above into a single line).	[reply] [d/l] [select]
Re^2: unknown character in between text by Anonymous Monk on Sep 17, 2011 at 13:00 UTC
eeew :p `s/([^\000-\200])/'&#'.ord($1).';'/ge s/([^\000-\200])/sprintf '&#x%X;', ord $1/ge` [download]	[reply] [d/l]
Re^3: unknown character in between text by DanielSpaniel (Scribe) on Sep 17, 2011 at 13:05 UTC
:-) That looks tidier! Haven't tested it, but I get the gist ... I knew someone would be able to condense into a handful of bytes! But why 200 in the character class? I'm sure there must be a good reason, but I think I am more familiar with just seeing \000-\177	[reply]
Re^4: unknown character in between text by Anonymous Monk on Sep 17, 2011 at 13:54 UTC
Re^5: unknown character in between text by Anonymous Monk on Sep 17, 2011 at 13:56 UTC
Some notes below your chosen depth have not been shown here


There's more than one way to do things
	PerlMonks