Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Get Title ip's list browser

by alaa21a (Initiate)
on Oct 29, 2017 at 12:02 UTC ( #1202275=perlquestion: print w/replies, xml ) Need Help??
alaa21a has asked for the wisdom of the Perl Monks concerning the following question:

Hello whord, i have this code but its problem i want check file list ips and check http & https and i want all string Title
#!/usr/bin/perl use Parallel::ForkManager; use LWP::UserAgent; use LWP::Protocol::https; use Net::SSL; my @foundip; my $save = "result.txt"; my $i = 0; my $forkmanager = new Parallel::ForkManager->new("20"); $forkmanager->run_on_finish(sub { # must be declared before first ' +start' my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data) = @ +_; $out{ $data->[0] } = $data->[1]; #print "[onfinish]". $data->[0] ."\n"; if (($data->[1] == -1) or ($data->[1] == 1)) { push(@foundip,$data->[0]); } }); my $foundstring; my $cnt =0; my $match_found = 0; open($foundipFILE, "< ip"); while (<$foundipFILE>) { $line = $_; $line =~ s/\x0a//g; if (!grep {$_ eq $line} @foundip) { $cnt++; my $pid = $forkmanager->start and next; alarm("20"); $rez = 0; $rez = do_shit($line); #print "$cnt\r\n"; $forkmanager->finish(0, [ $line, $rez ]); # +Child exits }else{ #print "!!!!! $line was here\n"; } } close($foundipFILE); $forkmanager->wait_all_children(); print "\n!!!!! FINISH !!!!!!\n"; exit; sub do_shit{ my $line = shift; my ($ip,$port) = split (/:/, $line, 2); $link = "https://$line"; #print "$link\n\r"; $link = "http://$line"; print "\e[31mScaning $link\r\e[0m"; #start here my $browser = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 } +,); $browser->timeout(60000); my $response = $browser->get("$link"); my $head = $response->as_string(); #print $head; @types = ('google', 'yahoo', ); foreach(@types){ #print $type; if($head =~ m/(.*)$_(.*)/i) { #search for the key word print "\e[33mFound $link $_\r\n\e[0m"; open(OUTPUT,">>$save"); print OUTPUT "$link $_\r\n"; print OUTPUT "###################\r\n"; close (OUTPUT); } else { next; } } }

Replies are listed 'Best First'.
Re: Get Title ip's list browser -- suggestions for a newcomer
by Discipulus (Monsignor) on Oct 30, 2017 at 12:38 UTC
    Hello alaa21a and welcome to the monastery and to the wonderful world of Perl!

    So your post go unanswered.. why? it seems to me because it appears like Frankenstein code, build up cutting and pasting other's code..

    So, ok we approach Halloween, but Frankestein posts are not so much welcome here.. ;=)

    But since is your first post..

    I say this not to blast you down but to help you to learn better and ask better questions in the future: see How (Not) To Ask A Question for more details on how to ask something in a better way.

    First of all use strict; use warnings lines are missing from your code and part of it does not compile with them in force.

    Other parts will compile with use strict; use warnings and this let me to suppose you are cutting and pasting code.

    In adition always use the 3 arg form of open and even better use another variable for the filename and use lexical filhandle: infact your open(OUTPUT,">>$save") is considered sumerian nowadays..

    Take your free (yes; free!!) copy of the wonderful book by chromatic Modern Perl and read it.

    Now about your goal: before going to parallelize the HTTP(S) requests using Parallel::ForkManager start with something simpler like my code below. In addittion LWP::UserAgent is not thread safe and this is the rationale for LWP::Parallel and LWP::Parallel::UserAgent existence.

    So you want check an IP list for both HTTP and HTTPS connections? an get titles? As side note are you aware that an IP can have multiple websites configured on it to resposnd to an HTTP request (https sites are different)? you also know that a domain name can, by other hand, resolve multiple IPs?

    # ALWAYS use the below two lines!! use strict; use warnings; # ALWAYS use the above two lines!! use LWP::Protocol::https; use LWP::UserAgent; # also use the 3 form open and check for failures: my $ip_file = '/path/to/file.ext'; # use lexical filehandles # UPDATE ooops.. missed a comma.. see hippo below http://www.perlmonks +.org/?node_id=1202382 open $ip_fh, '<', $ip_file or die "Unable to open $ip_file for reading +!"; my $ua = LWP::UserAgent->new; while (<$ip_fh>){ # remeber to chomp lines arriving chomp; # get every IP my @ip = split /\s+/,$_; # process them foreach my $ip(@ip){ # construct the request for HTTP protocol my $httpr = $ua->get('http://'.$ip); # note that you just need title not the whole GET print "http://$ip\t\t",$httpr->status_line,"\t\t",$httpr->ti +tle,"\n"; # do the same for HTTPS protocol my $httpsr = $ua->get('https://'.$ip); print "https://$ip\t\t",$httpsr->status_line,"\t\t",$httpsr- +>title,"\n"; } } # sample output http://216.58.198.36 200 OK Google https://216.58.198.36 500 Can't connect to 216.58.198.36:443 http://66.39.54.27 200 OK PerlMonks - The Monast +ery Gates https://66.39.54.27 500 Can't connect to 66.39.54.27:443

    PS ..oops missed a comma in the filehandle opening: see hippo below

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
      hi Discipulus , if can help to me for run this code please
      #!/usr/bin/perl use Parallel::ForkManager; use LWP::UserAgent; use LWP::Protocol::https; use Net::SSL; my @foundip; my $save = "result.txt"; my $i = 0; my $forkmanager = new Parallel::ForkManager->new("2000"); $forkmanager->run_on_finish(sub { # must be declared before first ' +start' my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data) = @ +_; $out{ $data->[0] } = $data->[1]; #print "[onfinish]". $data->[0] ."\n"; if (($data->[1] == -1) or ($data->[1] == 1)) { push(@foundip,$data->[0]); } }); my $foundstring; my $cnt =0; my $match_found = 0; open($foundipFILE, "< ip"); while (<$foundipFILE>) { $line = $_; $line =~ s/\x0a//g; if (!grep {$_ eq $line} @foundip) { $cnt++; my $pid = $forkmanager->start and next; alarm("20"); $rez = 0; $rez = do_shit($line); #print "$cnt\r\n"; $forkmanager->finish(0, [ $line, $rez ]); # +Child exits }else{ #print "!!!!! $line was here\n"; } } close($foundipFILE); $forkmanager->wait_all_children(); print "\n!!!!! FINISH !!!!!!\n"; exit; sub do_shit{ my $line = shift; my ($ip,$port) = split (/:/, $line, 2); $link = "https://$line"; #print "$link\n\r"; $link = "http://$line"; print "\e[31mScaning $link\r\e[0m"; #start here my $browser = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 } +,); $browser->timeout(60000); my $response = $browser->get("$link"); my $head = $response->as_string(); #print $head; if($head =~ m/(.*)$_(.*)/i) { #search for the key word print "\e[33mFound $link $_\r\n\e[0m"; open(OUTPUT,">>$save"); print OUTPUT "$link $_\r\n"; print OUTPUT "###################\r\n"; close (OUTPUT); } else { next; } }
        hi Discipulus i'm tested your code but i have problem show this massege
        Global symbol "$ip_fh" requires explicit package name at ./run.pl line + 15. Missing comma after first argument to open function at ./run.pl line 1 +5, near "$ip_file or" Global symbol "$ip_fh" requires explicit package name at ./run.pl line + 19. Execution of ./run.pl aborted due to compilation errors. +

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1202275]
Approved by davies
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (8)
As of 2017-12-13 12:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What programming language do you hate the most?




















    Results (361 votes). Check out past polls.

    Notices?