Re^3: ithreads picks fight with LWP::Agent; everybody loses (Works for me!)

[BrowserUk]

Perhaps he could try reverting to v1.67 (the version I have that works, albeit on a win32 build).

He might also try the following and see if it completes clean (and post the output if not):

#!/usr/bin/perl -w
use POSIX qw[ _exit ];
use Time::HiRes qw[ usleep ];
use threads ( stack_size => 4096 );
use threads::shared;

$|++;

use HTTP::Request;
use LWP::UserAgent;

my $started :shared = 0;
my $ended   :shared = 0;

my $monitor = async {
    printf "\rStarted: $started Ended: $ended\t" 
        and uleep 100 while $ended < 500;
    print "\nAll done";
};

my $TEST_URL = "http://www.blairhippo.com/";

threads->create( 'execute_request', $TEST_URL )->detach for 1 ..500;

$monitor->join;

_exit( 0 );


sub execute_request {
    { lock $started; ++$started; }
    my $url = shift;
    my $request = HTTP::Request -> new ('HEAD', $url);
    my $agent = LWP::UserAgent -> new();
    my $response = $agent -> request($request);
    { lock $ended; ++$ended; }
}

[BlairHippo]

Thanks for the code and for your help. You raise a couple of points.

First, the premature thread termination -- yeah, I spotted that. In the real code this failure script is based off of, I do indeed take care to make sure all threads have finished before termination, so I was quite certain that wasn't the source of my trouble and left it out of this code. (Besides, on my server, the code seg faults before it ever becomes an issue.) Sorry, I should have made that clearer.

Second, your code. I had to tweak it slightly to run (had to set stack size to 4*4096, and s/uleep/usleep/), but once it ran, it gave me a more or less instant seg fault, though in much less detail than I'm seeing in my own code:

Started: 0 Ended: 0     Segmentation fault

Though the "instant segfault" feature should prove valuable to my debugging, since it beats waiting for my own code to finally get around to deciding to die.

(Whoops, I just realized I never specified that. In both my failure script and the real code, the crash isn't instantaneous; it runs a little while first. Sorry, that was relevant and should have been mentioned in my initial posting.)

Finally, thanks for the 1.67 recommendation; I'll definitely give that a try.

[BrowserUk]

(had to set stack size to 4*4096, and s/uleep/usleep/)

Sorry about the uleep, I changed it in the browser from the Win32::Sleep I used here.

I'm surprised you needed to increase the stack size? Your threads don't do much that would consume stack. Or is that a requirement of a 64-bit processor?

The main change I made in that version, beyond the logging, was to bypass perl's cleanup. It is usually during this phase that the mysterious, and for the most part inconsequential, "free to wrong pool" errors usually occur. Hence the reason you have to wait so long.

The other change was to use shared variables. I suspect that it is this that is the cause of the premature segfault. You could try omitting them and see if bypassing the cleanup with POSIX::_exit(); helps.

The problem here is that if you join the threads, the cleanup is triggered early and the problem will arise.

If you detach them, your main thread has nothing to wait on. That's what I used the shared vars and monitor thread to achieve.

Generally, I don't expect that iThreads have been exercised much on 64-bit builds.

Finally, I wouldn't recommend threads or forks for this type of usage. Both are to heavy for this purpose. An IO::Select or a LWP::Parallel will use far less resources for too much greater effort. Best of all, they are more likely to work in your environment.