Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^4: Mixed character encoding issues

by ddaupert (Initiate)
on Jul 11, 2012 at 02:16 UTC ( #980990=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Mixed character encoding issues
in thread Mixed character encoding issues

nikosv: I took your suggestion to export from excel as a csv file, and that did help a great deal. I want to thank you for that. It gave me a solid place to move from. By reading the csv file and writing it immediately out as a first step, I was able to verify the characters were like for like. Then I stacked on the additional steps of importing into the DB, etc., and watched what happened.

It was also quite useful to understand the bit about double encoding. I noticed as I made code changes that the data became more or less mangled. I found something quite amazing by watching this behavior. When creating my tables in SQLite, I used DBI more or less directly; I had been using the sqlite_unicode connection setting, but found I needed NOT to set sqlite_unicode => 1 in the connection statement. But contrary to that, when running queries through DBIx::Class machinery, I DID need to set sqlite_unicode => 1 in the DB connection. Once all that was sorted out, all data were read, input into the DB, read back out of the DB, and written out to the final files while preserving proper encoding.

Many, many thanks.

/dennis


Comment on Re^4: Mixed character encoding issues
Re^5: Mixed character encoding issues
by nikosv (Hermit) on Jul 11, 2012 at 14:43 UTC
    I'm glad I could help

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://980990]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2014-09-02 03:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (18 votes), past polls