Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: Perl+PostgreSQL+GeoIP = Awesome!

by erix (Parson)
on Nov 21, 2018 at 14:52 UTC ( #1226131=note: print w/replies, xml ) Need Help??


in reply to Perl+PostgreSQL+GeoIP = Awesome!

NICE indeed. But TIMTOWTDI and here's one more (not better, just shorter and how I would probably have done it)

(bash + perl, Pg connects via PG* envvars)

wget --timestamping http://geolite.maxmind.com/download/geoip/database +/GeoIPCountryCSV.zip wget --timestamping http://geolite.maxmind.com/download/geoip/database +/GeoIPv6.csv.gz echo " create table geoip ( netblock cidr not null, country_code text not null, country_name text not null, constraint geoip_pk primary key (netblock) );"| psql \ && ( unzip -p GeoIPCountryCSV.zip ; gunzip -c GeoIPv6.csv.gz ) \ | perl -MNet::CIDR -ne ' chomp; my @arr = split( /\"\,\ ?\"/, $_ ); my @cidr = Net::CIDR::range2cidr(sanitize($arr[0]) . "-" . sani +tize($arr[1])); for my $subcidr (@cidr) { print $subcidr, "\t", sanitize($arr[4]), "\t", sanitize($arr +[5]), "\n"; } sub sanitize { my ($val) = @_; $val =~ s/^\"//; $val =~ s/\"$//; return $val; } ' | psql -c "copy geoip from stdin with (format csv, header false, del +imiter E'\t');"

(I get 326604 rows (in 12 sec))

Replies are listed 'Best First'.
Re^2: Perl+PostgreSQL+GeoIP = Awesome!
by cavac (Curate) on Nov 23, 2018 at 10:00 UTC

    As just noted in the update/edit in my thread starter, i just found out that we need to switch to the new GeoLite2 databases, because the ones i have used here are no longer supported. I'll post a new version in CUFP after i get it working and fully tested.

    Otherwise, nice use of the psql COPY command ;-)

    perl -e 'use MIME::Base64; print decode_base64("4pmsIE5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCiAgTmV2ZXIgZ29ubmEgbGV0IHlvdSBkb3duLi4uIOKZqwo=");'

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1226131]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2019-03-22 03:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How do you Carpe diem?





    Results (110 votes). Check out past polls.

    Notices?