Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
We don't bite newbies here... much
 
PerlMonks  

Program unexpectedly terminates

by Anonymous Monk
on Nov 19, 2012 at 07:36 UTC ( #1004479=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a simple crawler that is based on LWP::UserAgent. It loops through a bunch of database records and fetches and processes web content related to each record. The problem is that the program will unexpectedly terminate at some point with not indication of the reason. This could happen a few minutes or a few hours into the processing. And when this happens the exit code of the program is 0. I am using Carp::Alway to enable stack-tracing and have added an END block as well as a DESTROY method to my crawler and neither of those produces any output when it terminates. I have also tried to override CORE::Global::exit. My code never calls exit and I'm positive I'm not accidentally short-circuiting my loop, otherwise the die after run would get called. Below is a skeleton of my code. Is there something else I can use to try and narrow this down?
#!/usr/bin/env perl use 5.016; use warnings; use Carp::Always; BEGIN { *CORE::GLOBAL::exit = sub { die } } END { say "END block"; } my $crawler = Crawler->new; $crawler->run; die; package Crawler { use parent 'LWP::UserAgent'; # use a bunch of other modules for the processing. sub new { my $self = $class->SUPER::new(%args); $self->{ids} = (1..100); # These normally are pulled from a D +B. return $self; } sub run { my $self = shift; my @idx = shuffle 0 .. -1 + @{ $self->{ids} }; my $cur = 0; for my $i (@idx) { my $id = $self->{ids}[$i]; printf "%d/%d: %s (%d)\n", ++$cur, 0+@idx, $id, $i; # Fetch and process stuff related to this id. } } sub DESTROY { say "Crawler destroyed" } }

Comment on Program unexpectedly terminates
Download Code
Re: Program unexpectedly terminates
by ColonelPanic (Friar) on Nov 19, 2012 at 08:31 UTC

    The code you have shown has a problem. You are assigning an array (1..100) as the value in a hash. You need an array reference for that (use brackets instead of parentheses). However, I doubt that is the real problem with your code--it was probably just a mistake when you created the simplified example.

    Also, I tried your CORE::GLOBAL::exit override, and it didn't do anything--it only got worked if I explicitly called the exit() command, which your code does not.

    I think your problem must be in the processing that you are not showing. It could be failing on the same one each time. Because you are processing in a random order, it would take varying amounts of time for that condition to be reached.

    I suggest some good, old-fashioned debug print statements within your processing loop. Log the id of each record you are looking at, and record the success of each significant operation. That should help you to know where it fails.



    When's the last time you used duct tape on a duct? --Larry Wall
      The code you have shown has a problem. You are assigning an array (1..100) as the value in a hash. You need an array reference
      Yeah, yeah. That's obviously not working code and I was trimming down my real program to just show the basic structure.
      Also, I tried your CORE::GLOBAL::exit override, and it didn't do anything--it only got worked if I explicitly called the exit() command, which your code does not.
      I said in my OP I never directly call exit. But my unstated assumption is that one of the many CPAN modules I am using might be calling it.
      I think your problem must be in the processing that you are not showing. It could be failing on the same one each time. Because you are processing in a random order, it would take varying amounts of time for that condition to be reached.
      I'm intentionally not wrapping anything in eval, so if something in my code does fail, it should bomb out loudly. It never does.
      I suggest some good, old-fashioned debug print statements within your processing loop. Log the id of each record you are looking at, and record the success of each significant operation. That should help you to know where it fails.
      The print statement I have in the example already prints before the processing starts, so I see which record it stops at. It's never the same one. Plus, as I mentioned above, any error in my code would die loudly. But I'm not sure why the program can terminate with exit value of 0, yet the END block is never called. So my assumption is that one of the CPAN modules is at fault, but whatever is the cause, I was hoping there was some module that would perform the necessary overrides and tell me what's calling exit.

        Thanks for the additional information. I didn't understand that you were trying to trap explicit calls to exit(). That makes sense, then.

        Playing around more with the BEGIN block, I found that an exit() statement within another package was only overridden if the BEGIN block came before the package definition. Do you have this block before you include all of your modules?

        Also, would Test::Trap be helpful?



        When's the last time you used duct tape on a duct? --Larry Wall
Re: Program unexpectedly terminates
by tobyink (Abbot) on Nov 19, 2012 at 09:29 UTC

    It would be helpful if the sample code you provided actually compiled (undefined $class, %args and &shuffle prevents compilation with strict subs/vars). Fixing those issues...

    ... the code seems to execute correctly here.

    perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
      Thanks for the helpful contribution Mr. Pedantic :-P

        Thank you for providing code that neither compiles nor illustrates your problem Mr Anonymous.

        perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
Re: Program unexpectedly terminates
by Anonymous Monk on Nov 19, 2012 at 12:40 UTC
    Anonymously speaking, the place to start is by adding print STDERR I am here now messages throughout the code. Watch for the last one.
Re: Program unexpectedly terminates
by aitap (Chaplain) on Nov 19, 2012 at 14:37 UTC
    Assuming you have an UNIX-like system, try running your program under strace. This will produce a lot of text, but probably last lines will show you the reason.
    Sorry if my advice was wrong.
Re: Program unexpectedly terminates
by Anonymous Monk on Nov 19, 2012 at 19:03 UTC
    As a shot in the dark, I added $SIG{PIPE} = 'IGNORE'; and haven't seen the problem reoccur in 7 hours, so it's looking like that solved the problem. Now I have to figure out if the culprit is LWP::UserAgent (and it's many dependencies) or Net::Proxy::Type, or IO::Socket::Socks. Then produce a repeatable test case and properly report it as a bug.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1004479]
Approved by philipbailey
Front-paged by philipbailey
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (17)
As of 2014-04-24 11:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (565 votes), past polls