(*)Updated units per pc88mxers post below.
By serialising the counts through a threaded udp server, batching them on input before transfering them to the background writing thread, I managed sustained throughput rates of close to 600k/minute on a single core 2.6 M*GHz machine running both producer and consumer, and with half the bandwidth taken up with a download going on simultaneously:
Consumer run:
c:\test>s-udp.pl -MAXBUF=500
Total: 347176 throughput: 9777/sec [max: 9987]
MAXBUF=500 seems to be about the sweet spot on my machine. I can get higher throughputs by upping the priority of the server process. I was just writing the counts out to a file in the background, but you could probably do batch updates to a db without slowing things much.
Consumer:
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Time::HiRes qw[ time usleep ];
use IO::Socket;
use List::Util qw[ max ];
$|++;
our $PORT ||= 9999;
our $FILE ||= 'theLog';
our $LEN ||= 6;
our $MAXBUF ||= 1000;
my $Q = new Thread::Queue;
open my $log, '+> :raw', $FILE or die "$FILE: $!";
my $start = time;
my $n :shared = 0;
my %log;
my $total = 0;
async {
my $max = 0;
my $thru = 0;
while( my $in = $Q->dequeue ) {
++$log{ $_ } for split chr(0), $in;
sysseek $log, 0, 0;
my $logRec = join "\n", map{ $_ . ':' . $log{ $_ } } sort keys
+ %log;
syswrite( $log, $logRec );
$max = max( $max, $thru = int( $n / ( time() - $start ) ) );
printf "\r\t Total: %8d throughput: %5d/sec [max:%5d]",
$total, $thru, $max;
$total += $n; $n = 0; $start = time();
sleep 1;
}
};
my $srv = IO::Socket::INET->new(
LocalPort => $PORT,
Proto => 'udp',
) or die "Socket: $@ : [$^E]";
my $buffer = '';
my $msg;
while( $srv->recv( $msg, $LEN ) ) {
next unless length $msg;
$n++;
if( length( $buffer .= chr(0) . $msg ) > $MAXBUF ) {
$Q->enqueue( $buffer );
$buffer = '';
}
}
Producer:
#! perl -slw
use strict;
use IO::Socket;
our $N ||= 1000;
our $PORT ||= 9999;
our $DELAY ||= 0.0001;
my $sock = IO::Socket::INET->new(
Proto => 'udp',
PeerPort => $PORT,
PeerAddr => 'localhost'
) or die "$@ [$^E]";
my $sent = 0;
for ( 1 .. $N ) {
++$sent;
$sock->send( int rand( 32767 ) );
select undef,undef, undef, $DELAY;
}
print "Sent: $sent";
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.