Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

perl crashes parsing huge script - how to find a line that crashes it?

by Anonymous Monk
on Feb 19, 2008 at 08:38 UTC ( #668751=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I've got a new job. There I have to support a huge perl script (6mb big) written by previous programmer; that script is used on Windows. It was coded for ActivePerl 5.8.x. Now due to various reasons I need to run this script with ActivePerl 5.10.x (previous programmer never attempted to run it with 5.10.x). The script does not use any modules besides shipped with ActivePerl.

The problem - perl 5.10 crashes parsing this huge 6mb script. I guess I need to tweak just a single line to make it work with new version of perl.

The question is - how to detect a line Perl crashes at.. Are there any commandline switches that can help me with that?

Note that Perl crashes during parsing, not during execution.

Thanks for your answers in advance!

Comment on perl crashes parsing huge script - how to find a line that crashes it?
Re: perl crashes parsing huge script - how to find a line that crashes it?
by Anonymous Monk on Feb 19, 2008 at 08:56 UTC
    I guess calling it like this: "perl -Dt script.pl" will sched some light. But this requires perl compiled with -DDEBUGGING. ActivePerl is not compiled with way.

    Does anybody know where to get such perl for windows?

    Thanks!

      Does anybody know where to get such perl for windows?

      I don't know of any pre-compiled debug versions of perl for windows, but it's quite easy to build one from source. Both the Win32/Makefile (for use with Visual Studio) and Win32/makefile.mk (for use with MinGW) in the perl source accommodate the building of a debug perl. (See the comments in the configurable section of the relevant makefile - it should just be a matter of uncommenting in one line.)

      Cheers,
      Rob
Re: perl crashes parsing huge script - how to find a line that crashes it?
by tilly (Archbishop) on Feb 19, 2008 at 09:02 UTC
    An old standby. Insert lines like the following:
    BEGIN {print "Still alive at " . __LINE__ . "\n"}
    and then do a binary search to find where it dies.

    When you find it, you will likely have to figure out why it is dying. It may well be due to a bug in Perl 5.10. Do not discount the value of trying to come up with the shortest program you can that crashes. Even if that program is meaningless to you, you can submit it as a bug report and someone else can try to fix it. Even more usefully though, you are likely to learn enough about the problem to figure out how to work around the bug.

    Good luck! (And hopefully you do not need to work with much of this previous programmer's code. 6mb and uses no modules???)

      I second that.

      Another idea would be to truncate the file and see if it doesn't crash. Do a binary search that way. Hopefully, using __END__ will do the trick. Or, use editor's undo feature.

Re: perl crashes parsing huge script - how to find a line that crashes it?
by igelkott (Curate) on Feb 19, 2008 at 09:23 UTC
    What's the error message? That probably reports where the problem lies.
      No error message from perl. I just get standard "application has crashed and was terminated" window.
        I just get standard "application has crashed and was terminated" window

        I think that indicates that the program has compiled successfully - and that you're getting a segfault (ie runtime error) before any other output has been produced.

        Some print "Got to here\n"; statements sprinkled throughout should help determine where the error is occurring.

        Cheers,
        Rob
        Is this some gui-application, or a commandline program started by clicking an icon? Anyway, try to start it from a Dos command shell, i.e. Start->Run->cmd
        look at the properties for the shortcut and cd to the right drive / directory accordingly. Starting from a commandline will leave you at the commandline with the possibility that you get some more information when it chrashes.
        hth
Re: perl crashes parsing huge script - how to find a line that crashes it?
by hipowls (Curate) on Feb 19, 2008 at 12:29 UTC

    Perhaps it is trying to load a module compiled for perl 5.8.x. To check at the head of the script put

    use Data::Dumper;
    and after each use statement put
    BEGIN { print 'use <module name> at line ', __LINE__, "\n"; print Dumper \%INC; }
    perl -V will show the paths searched by default for modules.

      and after each use statement put
      If the use call makes it crash, isn't this too late? In that case, the print statement should come before the use call.

      If it's not the use calls that makes it crash, then you can postpone the check of %INC until all modules are loaded, for example in a CHECK block, or else, at the top level at the very front of the script. Otherwise, you'll get a lot of nearly identical Dumper data for %INC, which is a lot of noise.

      In this case, I'd do this:

      #for every module (using Foo::Bar as an example): BEGIN { printf "use Foo::Bar at line %d \n", __LINE__; } use Foo::Bar; # replace with actual module name BEGIN { printf "Foo::Bar loaded OK\n"; } # once for the whole script: CHECK { use Data::Dumper; print Dumper \%INC; }
      If a use call makes it crash, you won't see the associated "module Foo::Bar loaded OK" message. If none did, you'll see the summary for %INC.

      I have no idea how you can print the path of a module if just use of it makes perl crash...

      p.s. As the OP says, no other than standard modules are used, then just a check of @INC might reveal if there's any suspicious root path in there.

      And, oh: don't just install ActivePerl 5.10 on top of the installation of ActivePerl 5.8. Either use a parallel intallation, in a different root directory, or completely uninstall perl first and install everything again. That way you can avoid using any old modules, compiled for AP 5.8.

Re: perl crashes parsing huge script - how to find a line that crashes it?
by graff (Chancellor) on Feb 19, 2008 at 13:47 UTC
    It is really hard to imagine how someone could accumulate 6 MB of content into a single perl script and keep it functional (presuming it actually does still function on a system that still uses 5.8.x), let alone maintainable (which, as you are discovering, it is not). No single script should ever get that big. (update: this consideration applies to source code in any programming language, not just perl)

    If you do still have a system running 5.8.x, and this script does still run (and does what it's supposed to do) on that system, I would recommend breaking the big script apart into smaller chunks, testing the "decomposed" version on the 5.8 system to make sure it produces the same results as the original, and port the multi-file version to the 5.10 system.

    I would expect that in the process of splitting it up, you will find numerous opportunities to refactor the code and eliminate lots of repetitive content. (One sure way of making a script too long is to use the "copy/paste/modify" style of coding, whereby a few lines of code get repeated dozens of times with minor alterations, instead of being written once as a proper subroutine with sensible parameters.) The app as a whole is likely to end up much smaller as a result, and that would be a Good Thing.

    One thing is for certain: if the code gets split into different source files, it will be a lot easier to isolate/locate problems.

    If the app needs to be around for a while, I would argue that this is the only sane and sustainable strategy -- don't propagate the mistakes of the programmer who left this mess behind. If/when it needs to be migrated again, it'll be easier at that time if you make the effort to refactor it now. Otherwise, you'll just have to face the same problem again later.

    One other thought: do you have access to a unix/linux box that you could try this out on? (Would it make sense to do that, or is the process really Windows-bound?) Options and tools for diagnosis, analysis and refactoring might be better in that other environment.

Re: perl crashes parsing huge script - how to find a line that crashes it?
by randyk (Parson) on Feb 19, 2008 at 17:20 UTC
    You can obtain the pdb debugging symbols for the standard ActivePerl builds at http://downloads.activestate.com/ActivePerl/Windows/. To use these, browse this site until you find the appropriate zip file containing the symbols (eg, ActivePerl-5.10.0.1002-MSWin32-x86-283697-symbols.zip), and unpack this in the same top-level directory where Perl was installed (eg, C:\Perl). Debugging tools like Dr. Watson, as well as Visual Studio if you have it, will then use the information in these symbol files to help pinpoint where and why a crash occurred.
Re: perl crashes parsing huge script - how to find a line that crashes it?
by dragonchild (Archbishop) on Feb 19, 2008 at 18:06 UTC
    First, I'm going to echo the "break it up" answer. If you're supporting this thing, you need to know what it does. That probably involves breaking it into pieces that work together. Me, I'm a fan of modules.

    A more immediate solution is the __END__ token. Stick it in the middle and see if everything works. If it does, then move it to 75%, otherwise move it to 25%. You'll very quickly determine about where the problem is.

    Another thing - write tests. Lots and lots of tests.


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Re: perl crashes parsing huge script - how to find a line that crashes it?
by sundialsvc4 (Monsignor) on Feb 19, 2008 at 22:05 UTC

    I am also going to echo the “break it up” admonition. And yes, I know that breaking up is ... well, you know the rest.

    Unfortunately, any programmer who built a 6-megabyte single source file never did know what (s)he was doing, but was going for a weird notion of job-security. In the long run, such a program is completely unmanageable, and it becomes a liability to the business. (This could be part of the reason why that person is now a “previous” programmer...)

    An arbitrarily-large Perl program can be subdivided into packages with just a little effort. As the file becomes smaller, it will be easier to do. The first step will be to try to map out the big chunks of the existing code which are not used constantly, separating them from the chunks that are common-to-all. A 6-megabyte monstrosity might contain many megabytes of code that are almost never used:   such material can be required when ... well, you know ... when required! :-)

Re: perl crashes parsing huge script - how to find a line that crashes it?
by peterdragon (Beadle) on Feb 19, 2008 at 23:10 UTC
    Firstly, I can't believe it's 6MB.

    Anyway, you've run it with "perl -d" right? Did that show anything?

    Next thing I'd try is running perl within the Visual Studio debugger. You can download the "Express" edition from MS free. That will at least show you the subroutine / symbol where it's dying.

    But to be honest, when I've had this kind of problem before it's been pretty inscrutable, e.g. a problem with one Perl XS module causing crash errors elsewhere.

    GL, Peter
    http://perl.dragonstaff.co.uk

      Firstly, I can't believe it's 6MB.

      I can't either. My biggest single script is less than 20k, and my average is less than 3.

Re: perl crashes parsing huge script - how to find a line that crashes it?
by runrig (Abbot) on Feb 20, 2008 at 01:37 UTC
    Here's a barely tested script which may find more or less where the offending bits are:
    use File::Temp; my $file = shift @ARGV; my $low=1; chomp(my $high = `wc -l $file`); print "File is $high lines\n"; my %tried; while (my $next=int(($high-$low)/2)+$low) { last if $tried{$next}++; print "low: $low high: $high next: $next\n"; my $tmp = File::Temp->new(TEMPLATE => "tmp_XXXXXX", DIR=>"."); open(my $fh, $file) or die "Can't open $file: $!"; while (<$fh>) { print $tmp "__END__\n" if $. == $next; print $tmp $_; } close $fh; $tmp->close(); my $out = `perl -c $tmp 2>&1`; ( $out =~ /syntax OK/ ? $low : $high ) = $next; }
    It was tested on the following file (but not on Windows):
    use strict; use warnings; my $foo=1; BEGIN { die "Dead\n" } print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n"; print "hello\n";
    This assumes you have the unix command "wc" installed (which you can get if you install msys, though you can work around this in pure perl with a bit more code).

    Update: I can think of ways this will fail, e.g., if the __END__ line splits up a {...} block you'll get a syntax error which may not have anything to do with the problem you're looking for which will throw this method off...oh, well...back to the drawing board (more testing around the "perl -c" output for those kinds of errors might do the trick, but I'm all out of tuits).

Re: perl crashes parsing huge script - how to find a line that crashes it?
by Popcorn Dave (Abbot) on Feb 20, 2008 at 08:42 UTC
    In addition to the suggestions that other monks have given you, you might try the Perl Tk based Debugger. Perl Tidy may also be of benefit to you if you've got to wade through 6M of code.


    Revolution. Today, 3 O'Clock. Meet behind the monkey bars.

    I would love to change the world, but they won't give me the source code

Re: perl crashes parsing huge script - how to find a line that crashes it?
by Anonymous Monk on Feb 20, 2008 at 12:09 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://668751]
Approved by bingos
Front-paged by grinder
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (22)
As of 2014-07-29 15:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (219 votes), past polls