Perl Monk, Perl Meditation PerlMonks

by Random_Walk (Prior)
 on May 30, 2006 at 13:25 UTC ( #552486=note: print w/replies, xml ) Need Help??

Hi Jonny,

I have been working on exactly this problem lately and got a massive amount of help from BrowserUk++. Please see multithreaded tcp listener with IO::Socket for the discussion and some excellent code Re^7: multithreaded tcp listener with IO::Socket.

The main trick is in passing the file handles to a pool of threads that handle the connections and a little bit of trickery to stop them going out of scope too soon.

For my problem I have an architecture like this ...

1 main/listener Thread
listens for connections
passes socket via queue
cleans up old sockets
check data and send OK or ERROR to client,
instruction to close socket back to main thread
and puts ref to data on another queue for...
Pulls refs off the queue and decides which of several
handler threads will get it (may be one of more).
spawns handler if it is not yet running then puts
ref on queue for that handler.
do thing like logging, hearbeat monitoring
forwarding exceptions to a console etc etc.
Send periodic heartbeats through its own
mechanism to another monitor console so
we know it is still working
[download]

#### Update

I am also using this for performance monitoring, stats gathering etc. Not sure how varied your data sources are, we have everything from heartbeats and disk space monitors to robots running synthetic transactions on our live web sites. One big step in making it all work was to define a common format for the event data. We have based ours on IBM's "Common Base Event". There is a paper out there called "Cannonical Situation Data Format: The common base event". It makes the rest of the code much simpler.

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Replies are listed 'Best First'.
by bloonix (Monk) on May 30, 2006 at 19:01 UTC
Hi R.,

do you like to tell me how do you gathering the stats?
Are there any Perlmoduls you use?

Cheers,
Jonny

Hi Jonny

We have an eclectic mixture of scripts developed over some time. A lot are shell and Perl4, the more recent stuff is Perl5 but I really mean recent, when I started there less than six months ago it was all Perl4. I am trying to encourage use of modules but given the normal security/paranoia/big.org it is not easy to push anything outside core. It also all has to be cross platform (win/aix/linux mostly, some special stuff for mainframes but that is an accepted exception to X-platform rules).

The stuff we are mostly using outside of core are XML modules (we use J-meter to run synthetic transactions on web pages and it reports in XML) and various DBI modules.

from core we use the threading modules, IO::*, Time HiRes, Net::Ping, FindBin, File::Find and a handful more I can not recall without greping our source.

We are building a large library of our own modules, some very in house and some that I am trying to get on CPAN. Ironicaly management love open source (free as in beer) but when I proposed to put some of our modules on CPAN I hit a sort of sand trap of redirection and passive resistance. No one will say no but no one will approve it either.

So for the most part now it is writing wrappers for legacy code and working on the sexy new core system.

Drop me a /msg and we can discuss further withouth going to re^12 and filling up nodes with trivialities for others.

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Create A New User
Node Status?
node history
Node Type: note [id://552486]
help
Chatterbox?
 [Corion]: thezip: If you want to open vim and can live with opening a second console window, use start "The results" vim.exe c:\path\to\logfile .log [thezip]: Ooops... I lied. I guess Cygwin is back. I'll just do a tail -f instead. Better. Sorry for the noise.

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (13)
As of 2017-03-27 18:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
Voting Booth?
Should Pluto Get Its Planethood Back?

Results (321 votes). Check out past polls.