|Problems? Is your data what you think it is?|
Mixed character encoding issuesby ddaupert (Initiate)
|on Jul 05, 2012 at 22:20 UTC||Need Help??|
ddaupert has asked for the
wisdom of the Perl Monks concerning the following question:
I've had no dealings with encoding previously, and am running into a bit of complexity. This may be a bit long, so let me first summarize.
SUMMARY: There are multiple steps data passes through where encoding may come into play. The input step is different between production and testing, but all further steps are the same. Production will import data from a MSSQL Database, whereas testing will input data from Excel spreadsheets. For now, I am focused on the testing process. (Yes, it would be better to also import test data from MSSQL, but we are constrained.)
OUTLINE of the testing source-to-destination steps:
The input spreadsheet I have contains user data where names and addresses are from some European countries (I do not know which, nor what languages are in play.) I am running into garbled characters from the very first step. I am not sure how to inspect the data at each step to compare data along the way.
I believe Excel stores data using cp1252, so have tried to convert the data to utf8. The code below gives the error "Cannot decode string with wide characters" which makes me think there may be mixed-encoded data in my input; that is, maybe some columns are of different encoding than other columns.
Example input/output character mangling:
Some name in input xls spreadsheet: Toxv‘rdlater....
Same name in output csv spreadsheet: Toxvâ€˜rd