Re^2: Win32 - Memory can not be "read"

[HuckinFappy]

Thanks for all the ideas jbert. The machines are under relatively significant load, but not in terms of number of processes. These dual processor machines have the following on them:

• A perl client script (well, 2...one per processor)
• Client polls server to get next available build job
• Client spawns perl script (and then waits for it) which:
  1. converts make template into make file
  2. runs make
• client notifies server job is complete and gets next one

So there's not a lot going on in parallel, but the network/cpu/memory can get pretty hammered (the local disk is almost a noop in most of these cases)

The perl is slightly mroe complicated than you see in most build environments, but it's not rocket science by any means.

I was able last night to re-enable the pop-ups, and caught some data this morning. It's Greek to me, but I'll include it here for the sake of completeness. First, the last few frames of the stack trace (I have it all for anyone interested):

>msvcrt.dll!77c46fa3()     
perl58.dll!Perl_newFOROP(interpreter * my_perl=0x00225ffc, long flags=
+0, char * label=0x01867624, unsigned long forline=32, op * sv=0x00000
+000, op * expr=0x018956f4, op * block=0x018955dc, op * cont=0x0000000
+0)  Line 3877 + 0x9    C
perl58.dll!Perl_yyparse(interpreter * my_perl=0x0173adfc)  Line 257 + 
+0x18    C
perl58.dll!S_doeval(interpreter * my_perl=0x01890668, int gimme=0, op 
+* * startop=0x00000000, cv * outside=0x00000000, unsigned long seq=26
+30)  Line 2817 + 0x6    C
perl58.dll!Perl_pp_require(interpreter * my_perl=0x0167a5ac)  Line 331
+4 + 0x3a    C
perl58.dll!Perl_runops_standard(interpreter * my_perl=0x00225ffc)  Lin
+e 23 + 0xc    C

And the disassembly of the memory around where it faulted:

77C46F9B  and         edx,3 
77C46F9E  cmp         ecx,8 
77C46FA1  jb          77C46FCC 
77C46FA3  rep movs    dword ptr [edi],dword ptr [esi] 
77C46FA5  jmp         dword ptr [edx*4+77C470B8h] 
77C46FAC  mov         eax,edi 
77C46FAE  mov         edx,3 
77C46FB3  sub         ecx,4

I did go look up op.c, which is where Perl_newFOROP() is defined, and found line 3877, which is:

    Copy(loop,tmp,1,LOOP);

As I say, it's Greek to me, but maybe someone sees something significant here?

Thanks to all, I know we're on the fringe of "is this a perl issue or a Windows problem", which means we're moving from something I know something about into the huge unknown for me (completely linux-centric)

[BrowserUk]

perl58.dll!Perl_newFOROP(      ??? Doing a for loop ???
    interpreter * my_perl=0x00225ffc, 
    long flags=0, 
    char *label=0x01867624, 
    unsigned long forline=32,  ??? Does this mean line 32 of the sourc
+e file ???
    op * sv=0x00000000, 
    op * expr=0x018956f4, 
    op * block=0x018955dc, 
    op * cont=0x00000000
) Line 3877 + 0x9    C 
    
perl58.dll!Perl_yyparse(                ??? parsing ???
    interpreter * my_perl=0x0173adfc
)  Line 257 + 0x18    C

perl58.dll!S_doeval(                    ??? Evaling the code ???
    interpreter * my_perl=0x01890668, 
    int gimme=0, 
    op * * startop=0x00000000, 
    cv * outside=0x00000000, 
    unsigned long seq=2630
)  Line 2817 + 0x6    C

perl58.dll!Perl_pp_require(              ??? Loading a module ???
    interpreter * my_perl=0x0167a5ac
)  Line 3314 + 0x3a  C

perl58.dll!Perl_runops_standard(
    interpreter * my_perl=0x00225ffc
)  Line 23 + 0xc    C

The stack trace suggests that the problem occurs when you are processing a for loop. And the assember instruction where the trap is occuring is the rep move ... which supports that.

From that, I'd hazard a wild guess that you have a for loop that running off the end of the data it is copying, and from the stack trace, it looks like that for loop is located in a module--maybe at line 32 of the file?

Like I say, that's mostly guesswork, but if your script doesn't have too many dependancies it might be worth looking at line 32 of each of them and seeing if there is a for loop there, before dismissing this completely :)

[HuckinFappy]

Wow...it's amazing what we learn on Perlmonks! :)

Thanks, I took your advice, and did some digging. Will you be too surprised to learn I found a for() loop on line 32 of a module?

It's not our module though...it's Exception::Class. Here's the beginning of the code in that module (ending at the for() statement on line 32):

package Exception::Class;

use 5.005;

use strict;
use vars qw($VERSION $BASE_EXC_CLASS %CLASSES);

BEGIN { $BASE_EXC_CLASS ||= 'Exception::Class::Base'; }

$VERSION = '1.19';

sub import
{
    my $class = shift;

    local $Exception::Class::Caller = caller();

    my %c;

    my %needs_parent;
    while (my $subclass = shift)
    {
   my $def = ref $_[0] ? shift : {};
   $def->{isa} = $def->{isa} ? ( ref $def->{isa} ? $def->{isa} : [$def
+->{isa}] ) : [];

        $c{$subclass} = $def;
    }

    # We need to sort by length because if we check for keys in the
    # Foo::Bar:: stash, this creates a "Bar::" key in the Foo:: stash!
 MAKE_CLASSES:
    foreach my $subclass ( sort { length $a <=> length $b } keys %c )
    {

The only place we do a 'use Exception::Class', it is accompanied by a long list of our own classes of Exception we define.

again, I'm not sure what this means, but it feels like a red herring, since this code works 99.99% of the time and is exercised tens of thousands of times per day.

I see op.c got a mountain of work between 5.8.5 and 5.8.8, so now I'm hoping to get approval for testring 5.8.8 to see if the issue clears up. It's still odd that this suddenly became an issue, but perhaps some change in Windows (security patch?) aggravated a known issue in op.c?

Thanks again!

[BrowserUk]

I'd stick a trace statement (probably to a separate file), either side of that for loop.

MAKE_CLASSES:
printf $temporaryLog "%s : Enter loop %d keys ...", scalar localtime()
+, scalar keys %c;
    foreach my $subclass ( sort { length $a <=> length $b } keys %c )
    {
     ...
    }
printf $temporaryLog "exited\n";

And run that on whichever machine is giving the greatest frequency of failures. At worst, it will allow you to eliminate a red-herring. At best, it just might show you something that will take you a step closer.

In a similar vein, if you have a test machine that is demonstrating the failure regularly, or a production machine that you can sign off on having run really slowly for a while, then get Devel::Trace and use it to see where the failure is occuring.

perl -d:Trace yourscript.pl 2>yourlog

The problem with that is the logfile will be overwritten by subsequent successful runs, If you use 2>>log you might fill your disc, and you'd have a mountain of log to search through.

So, add a couple of lines to the script:

BEGIN{ close STDERR; open STDERR, '>', "$$.log" }
END{ close STDERR; unlink "$$.log"; }

That will direct the trace output to a separate file for each run and, if the program completes successfully, delete that file and prevent your disk from filling up. Any runs which terminate abnormally should leave their log files behind which should give you a pretty clear picture of where the script is terminating.

Good luck!

[mr_mischief]

This appears to be related to this ActiveState bug report against ActivePerl 809 (built on 5.8.3) and also this Perlbug ticket for 5.8.0 on Linux. It's listed as unconfirmed and medium priority by ActiveState. It's stalled waiting for example code that triggers it on perlbug.

There seems to be a pretty good explanation of what appears to be going on (although I can't vouch for its accuracy). It seems from the two separate bug reports against two different dot releases on two different platforms that the bug came from upstream of ActiveState and affects at least four dot releases of 5.8 so far. I don't see anything in change logs saying it's been fixed in newer versions, but I'll admit I may have missed it. Also, it may be lumped into one of the "many things fixed" lines somewhere in a perldelta.

I can't find an ActivePerl built from 5.8.5 so I guess it's pretty certain you're using something else. Another Perl on Windows project or built from source? On cygwin or not?

Since the bug reports are waiting for an example or more of the problem, I'd suggest replying to stmpeters on the Perl 5 RT system at the above-mentioned bug #34450 about your code. If it turns out to be a known issue fixed in a newer version, at least the bug tracker will be updated to show that and you'll be told what version was first fixed.

Of course, this being PerlMonks, someone with the skills, time, and access to look into this might be reading the thread right now. I wouldn't hold my breath waiting for that, though.

Might be good to test a newer version first. If a test box with 5.8.7 or 5.8.8 doesn't seem to fix it, you might consider a bug report after that.

[HuckinFappy]

Thank you! +++++++ I'll see if I can somehow boil this down to a test case...if not, at least it's nice to understand what's happening a little better!

[jbert]

If the problem is the for() bug referred to (and it seems like a good candidate) and a later version of perl doesn't have it fixed, then you could work around the problem by hacking on the module - you could try changing the for() loop to a while loop with an explicit counter on number of iterations.

[Steve_p]

I you have some example code, please please please reply to the ticket so I can reopen it. Its exeptionally hard to tell whether a bug is fixed or not without some code to test with.