The exiting (production) code uses the open pragma.
use open qw/:std :encoding(utf8)/;
From what I tell, that forces all $fh to UTF-8. After that, I found all hex-type substitution fail. As I've been labouring with the intention of substituting, I've been testing with '<:raw' instead.
But it looks like the encode step is what's needed. I can at least encode_json now.
use v5.22;
use warnings;
use Devel::Dwarn;
use Cpanel::JSON::XS;
use Encode qw/ decode encode /;
use Text::CSV_XS;
my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 1 });
while (my $line = <DATA>) {
say "Before=".$line;
my $string = encode('UTF8', $line);
my $x = $csv->parse($string);
warn $x if !$x;
my @data = $csv->fields;
Dwarn \@data;
my $structure = encode_json(\@data);
}
__DATA__
IR_Choix_0092.tif,C,psi,,"Wild pansy (Viola tricolor), 19th century il
+lustration","19th-century hand painted illustration of wild pansy, he
+art<92>s ease, or love in idleness (Viola tricolor) flower by Pierre-
+Joseph Redoute (1759-1840). Published in Choix Des Plus Belles Fleurs
+, Paris (1827).",N/A,"Pansy, pansies, wild, Viola tricolor, 19th cent
+ury, painted, Engraving, illustration, nobody, no-one, flower, artwor
+k, Pierre Joseph redoute, bloom, blossom, botanical, botanist, bud, f
+lora, floral, history, historic, horticulture, leaves, petal, petals,
+ plant, vintage, watercolor, flower head, painting, stem, victorian s
+tyle, botanic, flowers, plants, Botany",,C,Fl,N/A,,,,^M
A big thank you. It's been a very frustrating day.
Dermot
