|Keep It Simple, Stupid|
Perl and Postgresql: Invalid byte sequence for encoding "UTF8"by StoneTable (Beadle)
|on Dec 21, 2006 at 20:19 UTC||Need Help??|
StoneTable has asked for the wisdom of the Perl Monks concerning the following question:
Here's one that's been a persistent thorn in my side.
I'm trying to import some of my apache logs for statistical analysis. I get various bits of UTF-8 in there, either in the URI or the user-agent field. I've tried various hacks, such as running the text through tr, but I've never managed to solve this cleanly.
My locale is defined as UTF-8.
The error points to the text containing invalid UTF8 characters. What I'd like to do is truncate any invalid UTF8 characters, but I'm not sure how to go about that.
I could ignore the errors, but I'm trying to batch insert the data. If it hits one of the invalid byte sequences, the entire transaction is rolled back. Not good, obviously.
What ways have you found to best handle/strip invalid UTF8 characters?