Carp::Heavy panic: malloc Unicode?

[Dave Howorth]

Om mani padme hum ...

I'm debugging a program and am seeing this message:

panic: malloc at /usr/lib/perl5/5.8.6/Carp/Heavy.pm line 61.
Debugged program terminated.

The code at line 61 of Carp::Heavy is:

  $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;

and it is preceded by a comment:

  # The following handling of "control chars" is direct from
  # the original code - I think it is broken on Unicode though.
  # Suggestions?

The whole module of course starts with the legendary words:

  heavy machinery, no user serviceable parts inside

I've searched around a bit without finding anything about my problem. Neither my code nor data does anything with Unicode; it's all ASCII AFAIK. I'm running perl v5.8.6. Can anybody suggest an approach to this problem?

Thanks, Dave

PS I see that the latest version of Carp::Heavy on CPAN has slightly different code that mentions utf8, but it still has the same comment!

[Steve_p]

That bit of code is broken and should be available in a newer version of Perl. The patch below resolved the problem.

Change 25363 by rgs@bloom on 2005/09/07 11:09:10

    The formatting function of Carp::Heavy has problem with utf8 strin
+gs.
    Work around it.

Affected files ...

... //depot/perl/lib/Carp/Heavy.pm#24 edit

Differences ...

==== //depot/perl/lib/Carp/Heavy.pm#24 (text) ====

@@ -116,9 +116,10 @@
   $arg = "'$arg'" unless $arg =~ /^-?[\d.]+\z/;
 
   # The following handling of "control chars" is direct from
-  # the original code - I think it is broken on Unicode though.
+  # the original code - it is broken on Unicode though.
   # Suggestions?
-  $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
+  utf8::is_utf8($arg)
+    or $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1)
+)/eg;
   return $arg;
 }

[graff]

Wow. That piece of enlightenment gives me hope that someday I might be able to use the perl debugger with unicode data... I've had repeated failures when using "perl -d" on scripts that involve regex operations on utf8 strings, in 5.8.6 on macosx and 5.8.7 on freebsd. A script would run okay by itself (except for bugs), but running it with "perl -d" leads to various reports of "out of memory" at the first unicode-heavy regex. So, I'm guessing the debugger contains something similar to the logic you've cited in Carp::Heavy. (But curiously, I can run a one-liner applying that regex to a utf8 string on my mac without difficulty. I'm confused.)

Anyway, I can't say that I approve of the approach taken in that patch. All it's doing is: if the string has the utf8 flag on, avoid trying to make any non-visible or non-ascii characters "displayable" by converting them to hex notation.

Rather than just give up completely on utf8 strings, wouldn't it be better to use a different approach? e.g.:

$arg = join "", 
       map {
             ($_ lt ' ' or $_ gt '~') ? sprintf("\\x{%x}",ord) : $_ 
           } split //, $arg;

I'm not sure, but I think that should avoid the malloc explosion caused by using [[:cntrl:]]|[[:^ascii:]] on a utf8 string.

[shmem]

Perl croaks in Perl_malloc (in malloc.c). perldiag states:

panic: malloc

(P) Something requested a negative number of bytes of malloc.

It would be interesting to know what's in $arg just before the crash. Maybe insert a print statement at line 60 into Carp::Heavy?

[jmcnamara]

I can't say if it is the case here but any time that I've encountered Carp::Heavy errors/warnings it has been a problem not with Carp::Heavy but with the parent program consuming too many system resources such as memory or open file handles.

Update: Looks like it was a Carp::Heavy problem this time. See Steve_p's answer below.

--
John.