Great one, I'm happy to have inspired You :).

I used (some of) the code in my project, I hope to publish it soon.

Here's my small script that's using aggregate to, well, aggregate prefixes. That's not perl, I hope it's not a blasphemy here :) . And it actually uses zsh syntax, but it should not be a problem to bashify it.

#!/usr/local/bin/zsh

INDEX=1
while [ $INDEX -lt 70000 ]
do
        PREFIXES=`grep "[[:space:]]$INDEX$" example|awk '{print $1}'`
        if [ $PREFIXES ]
        then
                a=`echo $PREFIXES|aggregate`
                echo $a|awk '{print $1, "\t'$INDEX'"}'
                let INDEX=$INDEX+1
        else 
                let INDEX=$INDEX+1
        fi
done
[download]

For me it crunched ~220000 prefixes to ~120000 so I think it's worth to use it. Not only will the data be smaller but also there will be less lookups (it only aggregates data prefixes of a single AS, so if there are overlapping prefixes with different AS numbers they will be preserved).