Great one, I'm happy to have inspired You :).
I used (some of) the code in my project, I hope to publish it soon.
Here's my small script that's using aggregate to, well, aggregate prefixes. That's not perl, I hope it's not a blasphemy here :) . And it actually uses zsh syntax, but it should not be a problem to bashify it.
#!/usr/local/bin/zsh
INDEX=1
while [ $INDEX -lt 70000 ]
do
PREFIXES=`grep "[[:space:]]$INDEX$" example|awk '{print $1}'`
if [ $PREFIXES ]
then
a=`echo $PREFIXES|aggregate`
echo $a|awk '{print $1, "\t'$INDEX'"}'
let INDEX=$INDEX+1
else
let INDEX=$INDEX+1
fi
done
For me it crunched ~220000 prefixes to ~120000 so I think it's worth to use it. Not only will the data be smaller but also there will be less lookups (it only aggregates data prefixes of a single AS, so if there are overlapping prefixes with different AS numbers they will be preserved).