Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Perl 5.8.2 thread is much worse

by pg (Canon)
on Dec 25, 2003 at 21:00 UTC ( #317011=perlmeditation: print w/replies, xml ) Need Help??

I downloaded 5.8.2 yesterday, and after tried for one day, most, if not all, of my multi-threading programs are now broken. Those programs are stable with 5.8.1.

As a maintainance release, 5.8.2 does not offer much new stuff, and you can confirm this by reading its perldelta. According to perldelta, there are only two core enhancements:

  • hash randomisation, to most of our monks this is not important any way
  • threading, fixed some problems, but introduced some new problems that I consider even worse

I would suggest people not to upgrade, especially if you used thread in some of your main programs. This kind of situation might happen pretty often in the future, as they now changed their strategy and try to have maintainance releases more frequently:

  • 5.8.3 23:59:59 GMT, Wednesday December 31st 2003
  • 5.8.4 23:59:59 GMT, Wednesday March 31st 2004
  • 5.8.5 23:59:59 GMT, Wednesday June 30th 2004

Also, I suggest people to share your experiences with other monks once you downloaded a new version, and help others to decide whether to upgrade.

One of my borken program is a proxy I used to filter web content and speed up the connection. This program now core dumps almost every minute.

use threads; use Socket; use strict; use warnings; my $banned_type = { "cab" => 1, "class" => 1, "dat" => 1, "exe" => 1, "gif" => 1, "ico" => 1, "jpg" => 1, "js" => 1, "jsp" => 1, "png" => 1, "swf" => 1 }; my $banned_site = { "" => 1, "" => 1, "" => 1, "" => 1, "" => 1, "" => 1, "" => 1 }; my $trusted_site = { "" => 1, "" => 1 }; use constant RES_400 => "HTTP/1.1 400 Bad Request\r\n\r\n"; my $proto = getprotobyname('tcp'); socket(BROWSER_LISTENER, PF_INET, SOCK_STREAM, $proto) || die "Failed +to create socket: $!"; setsockopt(BROWSER_LISTENER, SOL_SOCKET, SO_REUSEADDR, pack("l", 1)) | +| die "Failed to setsockopt: $!"; bind(BROWSER_LISTENER, sockaddr_in(8080, INADDR_ANY)) || die "Failed t +o bind: $!"; listen(BROWSER_LISTENER, SOMAXCONN) || die "Failed to listen: $!"; print "Internet filter started\n"; while (1) { my $browser; accept($browser, BROWSER_LISTENER); my $req; my $chunk; do { } until (!sysread($browser, $chunk, 10000) || ($req .= $chunk) =~ +m/\r\n\r\n/); my $host = ($req =~ m/Host:\s*(.*?)\r/)[0]; my ($method, $page) = ($req =~ m/^(.*?)\s+(.*?)\s/); if ($host && $page) { if (is_trusted_site($host)) { threads->create(\&process_one_req, $browser, $req, $method +, $host)->detach(); print "requested [$page] from a trusted site\n"; } else { if (is_banned_site($host) || is_banned_type($page)) { print "[$host, $page] is banned\n"; print $browser RES_400; close($browser); } else { print "[$host, $page] is not banned\n"; threads->create(\&process_one_req, $browser, $req, $me +thod, $host)->detach(); } } } else { close($browser); } } sub process_one_req { my ($browser, $req, $method, $host) = @_; my $iaddr = inet_aton($host) || die "no host: $host"; my $paddr = sockaddr_in(80, $iaddr); $proto = getprotobyname('tcp'); my $remote; socket($remote, PF_INET, SOCK_STREAM, $proto) || die "socket: $!"; connect($remote, $paddr) || die "connect: $!"; print $remote $req, "\r\n"; my $chunk; while (sysread($remote, $chunk, 10000)) { print $browser $chunk; open(LOG, ">>log.txt"); print LOG $chunk; close LOG; } close($remote); undef($remote); close($browser); undef($browser); undef($req); undef($host); } sub is_banned_site { my $site = shift; return 1 if (exists($banned_site->{$site})); if ($site =~ m/offeroptimizer/) { return 1; } if ($site =~ m/$/) { return 1; } if ($site =~ m/$/) { return 1; } if ($site =~ m/$/) { return 1; } return 0; } sub is_trusted_site { my $site = shift; return 1 if (exists($trusted_site->{$site})); } sub is_banned_type { my $tmp = lc(shift); =document $tmp =~ m/(.*?)\:\/\/((?:(?:.*)\/)*)(.*)/; my $proto = $1; my $path = $2; my $file = $3; my $query; my $type; if ($file) { ($file, $query) = split(/\?/, $file); ($file, $type) = split(/\./, $file); } =cut $tmp = (split /\//, $tmp)[-1]; my $type = (split /\./, $tmp)[-1]; print "type = $type\n" if ($type); if ($type && exists($banned_type->{$type})) { return 1; } else { return 0; } }

Replies are listed 'Best First'.
Re: Perl 5.8.2 thread is much worse
by liz (Monsignor) on Dec 26, 2003 at 11:54 UTC
    Most importantly you didn't mention on which system you are doing this. Although threads provide the same API on every system Perl runs on, internally the implementations need to glue themselves to very different underlying OS features.

    From the fact that you just downloaded 5.8.2 (and did not compile) I would guess you're running this under a Win32 system. If it is an ActiveState build, please report your problem with ActiveState.

    If you are however running on a *nix system and compile perl's yourself, please compile one with debugging enabled (add -Doptimize='-g' to configure) and try to force the coredump to be saved to disk (try ulimit-c unlimited and run gdb perl corefile on the coredump file to get a backtrace. And report that backtrace back here, or to p5p.

    That would be helpful in fixing this problem before 5.8.3, of which the code freeze is Real Soon Now.


Re: Perl 5.8.2 thread is much worse
by Zaxo (Archbishop) on Dec 26, 2003 at 05:24 UTC

    Try running under perl -d so you can see which line cores. You didn't mention if any other error messages came up, but that is usually a sign that memory was freed too soon. Might be a refcount problem.

    After Compline,

      Thanks Zaxo, castaway and liz for replying. After more investigation, now I clearly believe that it is detach() now broken. The version I used is Win32, but the bug could well be across all platforms. Although as liz pointed out, the threading implementation is platform specific, I believe, most likely what broken is the common wrapper on top of the threading.

      Theoritically, there was no need for detach() in my code. I added them against my own will, as it was a must under 5.8.1 thread. Under 5.8.1, there was a huge memory leak without detach(), making it even worse, Perl didn't check null pointer returned from malloc, and after a while, once the memory leak is big enough, and the program can no longer allocate memory, the script just crashes.

      I tried to remove those detach(), and ran my script again, it seems fine, and no longer crashes like 5.8.1. But as I pointed out, now detach() core dumps.

      This is really a warning sign, and I am really concerned about the quality of, not just a particular release, but their development and verification process. To be frank, it is a big surprise to see that, they didn't carefully test some basic cases in the area, which they applied some big fixes.

      I am not worried about couple of bugs, but I can become really concerned if there are evidences showing that the way they control the quality of releases is flawed, and their verification (testing) process is flawed.

      We cannot control the quality of other people's work, but at least I can report the problem, and help them a little bit.

        So you had to remove a workaround for a no longer existent bug from your code. Sounds like an overall win to me. I don't see how you can talk about poor QA when the p5p folks explicitly mention that they're aware that the new code is problematic. Apparently they simply deemed the old code so bad that they preferred to toss out unfinished code for this fix-date release than keep the previous stuff.

        At least judging by the amount of investigation to prepare your initial post (none, you just complained "my scripts are broken now" - if you made any debugging attempt you certainly didn't mention it) I'd be more inclined to trust the p5p opinion.

        Update: I wasn't saying detach were a workaround per se, just that pg put it in "against his will" as he says because it was necessary.

        Makeshifts last the longest.

Re: Perl 5.8.2 thread is much worse
by castaway (Parson) on Dec 26, 2003 at 09:57 UTC
    As I mentioned before, I got 5.8.2 becaue it *fixed* a threads/core dump problem I was having in 5.8.1. I havent seen any other things that were broken in 5.8.2 and worked in 5.8.1. From your code its also very difficult to tell whats actually broken there, could you refine it to a smaller example?


Re: Perl 5.8.2 thread is much worse
by gmpassos (Priest) on Dec 28, 2003 at 07:04 UTC
    Just to comment about sysread(). If I'm not wrong, for sockets you can't try to read more than 1024*8 bytes per time with sysread()! At least with Perl-5.6.1 the bug exists. But since you are using the system resource I don't know if this is really a bug.

    Soo, for portability use read(), and you will ensure that you are reading all the data. (As documented in the Perl POD).

    Graciliano M. P.
    "Creativity is the expression of the liberty".

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://317011]
Approved by Corion
Front-paged by bart
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2018-11-20 21:52 GMT
Find Nodes?
    Voting Booth?
    My code is most likely broken because:

    Results (232 votes). Check out past polls.