Re: utf8::upgrade and $1

[ikegami]

The fix doesn't seem correct. Why would you want it to be a no-op for magical variables? Not even a warning!

Seems to me it should do a mg_get initially, upgrade the string, then a mg_set after the upgrading (taking care not to set SVf_POK).

It seems someone started to do it this way, given the presence of sv_utf8_upgrade_nomg. You might want to find the story behind that.

If you go down that route, the following fix is necessary in sv.h:

-#define sv_utf8_upgrade(sv) sv_utf8_upgrade_flags(sv, SV_GMAGIC)
+#define sv_utf8_upgrade(sv) sv_utf8_upgrade_flags(sv, SV_GMAGIC|SV_SM
+AGIC)

If not, that line still needs fixing.

[creamygoodness]

I agree, the fix is not finished yet. I suspect that the patch works because the rest of the function "mistakes" the private string for a public string, so to speak, and that the test suite passes because either the function is never invoked on a capture variable or because no variable with PERL_MAGIC_sv passes through that function which lacks a private string. Note that the patch doesn't affect all magic variables, just those with PERL_MAGIC_sv -- which is only one type among many, despite the name. From perl.h:

#define PERL_MAGIC_sv         '\0' /* Special scalar variable */
#define PERL_MAGIC_overload   'A' /* %OVERLOAD hash */
#define PERL_MAGIC_overload_elem  'a' /* %OVERLOAD hash element */
...

Nevertheless, it's useful to have isolated the problematic section of code.

I managed to hunt down some prior discussion: http://rt.perl.org/rt3//Public/Bug/Display.html?id=60472.

Nicholas Clark had this to say:

POK shouldn't be on for something with GMG. It should only be pPOK, so that any calls to SvPV() etc call into Perl_sv2pv_flags(), and the get magic is triggered.

(GMG is "get magic".) Based on Nick's post, I think we can confirm that the fact that $1 leaves this function with the SVf_POK flag on is a bug.

demerphq had this to say with regards to why $1 doesn't carry the READONLY flag:

Turns out that this is deliberate, as some pluggable regex engines allow for a writable $1.

Aevar patched this in quite some time ago.

So I think at this points its a bug in Encode, but I couldn't figure out how to fix it in the time I had available.

Perhaps we have tracked down this "bug in Encode".

Incidentally, this bug is a regession in Perl 5.10. Perl 5.8.8 doesn't have this problem.

[ikegami]

I think we can confirm that the fact that $1 leaves this function with the SVf_POK flag on is a bug.

Yes.

demerphq had this to say with regards to why $1 doesn't carry the READONLY flag:

Not a problem for my solution. The setter called via mg_set after the upgrade will throw an error if appropriate.

Turns out that this is deliberate, as some pluggable regex engines allow for a writable $1.

That means that upgrading $1 might have no effect or might upgrade the entire source string in those engines, but I don't see that as a problem. Side-effects are expectedintended when playing with magical variables.

Incidentally, this bug is a regession in Perl 5.10. Perl 5.8.8 doesn't have this problem.

That would explain the presence of sv_utf8_upgrade_nomg.