Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^2: Making this script process 56,000 lines 5 times faster

by marioroy (Priest)
on Mar 22, 2018 at 07:02 UTC ( #1211486=note: print w/replies, xml ) Need Help??


in reply to Re: Making this script process 56,000 lines 5 times faster
in thread Making this script process 56,000 lines 5 times faster

Hi kris004,

Another option is having Perl read the Curl output directly versus using LWP::Simple. The "-|" mode for open means the string that follows is interpreted as a command that pipes the output to us. See open function.

#!/usr/bin/perl use strict; use warnings; my $input = "https://raw.githubusercontent.com/StevenBlack/hosts/mast +er/hosts"; my $output = "ads.conf"; open my $IN, "-|", "curl $input" or die "open error $input: $!"; open my $OUT, ">", $output or die "open error $output: $!"; while ( <$IN> ) { if ( /^0\.0\.0\.0 (.*)$/ ) { print {$OUT} "local-zone: \"" . $1 . "\" redirect\nlocal-data: \"" + . $1 . " A 0.0.0.0\"\n"; } } close $IN; close $OUT;

Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1211486]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2019-03-23 08:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How do you Carpe diem?





    Results (114 votes). Check out past polls.

    Notices?