perlquestion creamygoodness Greets, In the Perl core, there's a function called <c>Perl_sv_utf8_upgrade_flags_grow</c>. Many paths ultimately lead to this function; I got there by way of the XS macro <c>SvPVutf8</c>, but the easiest way invoke it from Perl-space is via <c>utf8::upgrade</c>. It turns out that this function doesn't play nice with capture variables like <c>$1</c>. After it is invoked on them, they no longer capture properly: <c> marvin@smokie:~/perltest $ bleadperl dollar_one_utf8_upgrade.pl Without utf8::upgrade... a b c With utf8::upgrade... a a a Problem persists... a a a marvin@smokie:~/perltest $ </c> <readmore> Here's the test script: <c> use strict; use warnings; my $text = "a b c"; print "Without utf8::upgrade...\n"; while ( $text =~ /(\S)/g ) { print "$1\n"; } print "With utf8::upgrade...\n"; while ( $text =~ /(\S)/g ) { print "$1\n"; utf8::upgrade($1); } print "Problem persists...\n"; my $more_text = "d e f"; while ( $more_text =~ /(\S)/g ) { print "$1\n"; } </c> The problem appears to be that <c>Perl_sv_utf8_upgrade_flags_grow</c> turns on the <c>SVf_POK</c> flag by way of <c>SvPV_force</c>. It doesn't seem to have anything to do with whether the <c>SVf_UTF8</c> flag is on in either the string being regexed or the capture variable itself. Applying the following patch to sv.c in blead appears to kill the bug at the source. All of Perl's test cases still pass after it is applied. <c> marvin@smokie:~/projects/perl-git $ git diff diff --git a/sv.c b/sv.c index a53669a..280f064 100644 --- a/sv.c +++ b/sv.c @@ -3231,6 +3231,8 @@ Perl_sv_utf8_upgrade_flags_grow(pTHX_ register SV *const sv, const I32 flags, if (extra) SvGROW(sv, SvCUR(sv) + extra); return len; } + } else if (SvGMAGICAL(sv) && mg_find(sv, PERL_MAGIC_sv)) { + ; } else { (void) SvPV_force(sv,len); } </c> I'd like to solve this issue and supply a working patch via perlbug, so I can say that I solved a UTF-8 bug in the Perl core. :) However, I'm not sure that this patch is legit, because I don't understand exactly what <c>PERL_MAGIC_sv</c> is all about. I think what's going on is that $1 and friends are magical variables that should never have the <c>SVf_POK</c> flag on, since that indicates that they contain real strings. The regex engine probably doesn't use the standard string assignment interface and goes through the magic interface instead, hence its work is no longer visible once the standard channel is open. But is it safe to have the <c>SVf_POK</c> flag off for the remainder of <c>Perl_sv_utf8_upgrade_flags_grow</c>? <c>SvPV_force</c> was probably called for a reason, after all.