|Perl: the Markov chain saw|
Problem with utf8 after nearly 4096 bytesby gvieira (Initiate)
|on Sep 02, 2013 at 03:39 UTC||Need Help??|
gvieira has asked for the
wisdom of the Perl Monks concerning the following question:
Hello everyone, this is my first post here and it's blowing my mind for about a month.
I'm making an application with Perl/CGI who receives a file with a text, splits this text into words and compare them with a database. Those texts are in Portuguese and are part of a research in Linguistics, so the use of utf8 is essential.
I'm using the module utf8::all and he's wonderful... until nearly the byte 4096 of the text. I'm highlighting this cause I think that couldn't be a coincidence that the problem occurs exactly around 4kb. From this point to the end of the file, the perl becomes just incapable to recognize any accented letter and start to do some weird things like split the word on the accented letter like that: word (A)á(B) became words (A) and (B).
That's the part of the code that I think may be interesting. The whole code have almost 400 lines.
Any help would be great. This problem is disabling the whole research.