Re: Catching Cheaters and Saving Memory

[xdg]

It seems like you're structuring your data around users rather than around the pair-wise observations. Having a hashref for each user is costly. As someone else suggested, join up the user ID's into a single key. And since you only care about when both are in the same thread or both vote, you can sort them before you join them into a key.

sub join_ids {
  return = $_[0] < $_[1] ? "$_[0];$_[1]" : "$_[1];$_[0]";
}

Depending on the nature of your user id's, you could possibly pack() them rather than doing string concatenation.

Then, for every pair observation, you're tracking two counters: number of times together and number of votes for each other. Rather than keep two separate hashes (thus duplicating the storage of keypairs) or using an HOA, why not just pack() the two numbers in the data and then unpack them and update them each time you have an observation:

# assume my %obs for holding data;

my $zerozero = pack("LL",0,0);

for (my $i=0; $i<=$#current_user_ids; $i++) {
    for (my $j=$i; $j<=$#current_user_ids; $j++) {
        my $key = join_ids( $i, $j );
        my ($seen, $voted) = unpack( "LL", $obs{ $key } || $zerozero )
+;
        $seen++;
        if ($current_votes[$i]==1 && $current_votes[$j]==1) {
            $voted++;
        }
        $obj{$key} = pack( "LL", $seen, $voted );
    }
}

That gives you a single hash, with a fairly compact data representation (even more compact if you pack the IDs). It almost exactly matches your desired output, too.

If it's still too big to manage (if the number of user-pairs is high), I'd break it up by thread count -- e.g. 1 million threads per file. The output file would be userpairs with seen counts and vote counts and then you can just sum those. (It also parallelizes across machines nicely.)