Re^2: Encoding issue after upgrade

[Bod]

Provide a demonstration, such as a minimal program that exhibits the problem.

If I knew how to reproduce the problem, I wouldn't be asking for places to look for the problem!

In changing from one server setup to another, transferring the data from one instance of MariaDB to another via SQL dumpfiles, and running the same Perl scripts albeit under a different version of Perl, we have gone from correctly rendering webpages to webpages containing numerous  characters.

An example blog post on our test site. The  characters did not appear prior to the change.

[choroba]

Compare the database data bytewise (old versus new). Log each function's input arguments, compare the logs on the old and new systems. Run each function on the old and new system, compare the returned data (should have been covered by tests).

[Bod]

Compare the database data bytewise (old versus new)

Got it! Thank you.
**kicks self for not noticing sooner**

The rogue characters are all present in the database. So the problem was introduced during copying the data between database tables.

It probably needs to be manually cleaned up as there is more data added since the move...

[cavac]

It probably needs to be manually cleaned up as there is more data added since the move...

I would hazard a guess, that while the copy operation messed up the data, it messed it up in a very specific, methodical way. Probably all the bits of the data are still there (just in the wrong order/encoding) so with a bit of tinkering and math, you could possibly write a script to reverse the damage.

If you also have a copy of the old database, you could write a script that specifically updates the lines in the new database with the data from the old one, which would be even easier.

[Anonymous Monk]

check if anything changed in the DBD driver you are using. the data itself might not have been changed, just how it's decoded when fetched.

[ikegami]

If I knew how to reproduce the problem

You didn't say anything about the problem being intermittent. You made it sound like it was the opposite, that you always got the junk characters with previously-inserted data. Is this not the case? That would make the problem reproducible.

And since it is reproducible, the request is very straight forward. Simply remove everything that's not relevant. You can cut down on huge swaths of code by determining if it's a problem with the data coming from the DB, or if it's a problem with the output.

If it truly isn't reproducible, then please re-explain the problem more clearly.