Per the Text::CSV_XS docs, by turning on "binary", it allows the parser to accept a wider range of characters. "To cover the widest range of parsing options, you will always want to set binary."
I just was not making any assumptions about the data that you are trying to parse.
Given that, I'd say your next step would be to isolate which specific lines are failing. You can do this by writing the good lines into one file, and if possible, the bad lines into another file. If you can't write the bad lines into another file, then you'll have to determine which lines are missing from the good file.
To aid in this process, you might prepend a unique index (source line # ?) to the beginning of each line.
Where do you want *them* to go today?