Pass through perl http proxy server (no code, just advice)

[tachyon]

I have a need to implement an http filter ie Net Nanny/Surf Control style to protect innocent students from the evils of the internet at large. OK so I want to write a proxy server to accept requests, validate the URL and then do pass through/deny. I know how to do all of this with sockets, fork, redirects and a DBMS. The question is boring. Although I really love reinventing wheels I wish to learn from those who have gone before so any suggestions of what is currently available for me to R&D* would be appreciated.

cheers

tachyon

* R&D == Rip off && Duplicate. Open source == R&D && include attribution && share

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[merlyn]

I know you already know this, but for the onlookers, I wrote a few columns on that.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

[tachyon]

I just love getting you to say it! Kinda like Pavlov's dog and ringing that bell. Of course I have to grant it to you, been there done that, sometimes done it twice! Might make a good slogan for stonehenge....

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[BooK]

Well, uh, I'm just writing a module that's called HTTP::Proxy. ;-)

The goal is to be able to pass every request and response though a set of configurable Perl "filters".

Here's an example script that creates a anonymizing proxy, just like in merlyn's column (his script has been one of my inspirations, along with two others scritps by Abigail and Rafael Garcia-Suarez):

#!/usr/bin/perl -w
use HTTP::Proxy qw( :log );
use strict;

# a very simple anonymizing proxy
my $proxy = HTTP::Proxy->new;
$proxy->logmask( shift || NONE );    # log configuration

# the anonymising filter
$proxy->push_headers_filter(
    mime    => undef,    # apply this on any type of content
    request => sub {
        $_[0]->remove_header(qw( User-Agent From Referer Cookie ));
    },
    response => sub {
        $_[0]->remove_header(qw( Set-Cookie )),;
    },
);

$proxy->start;

(This code works with the development version.)

The proxy is not fully functional yet, though. One of my goal is to create a set of filters that would log the whole session, so that yet another script could (automatically) create a web robot from the session transcript.

If you are interested, there is a web site (older versions and CVS snapshot) and a mailing-list at http://http-proxy.mongueurs.net/. I am interested in help, and comments on the API (which is subject to change).

[zengargoyle]

not Perl and a heavyweight (functionwise, not necessarily slow/bloated) is Squid Web Proxy Cache which you could probably make do whatever you can imagine.

[IlyaM]

In my attempt to learn POE I wrote this simple single-process proxy:

#!/usr/bin/perl

use warnings;
use strict;

use POE qw(Component::Server::TCP Component::Client::HTTP
           Filter::HTTPD Filter::Stream);

POE::Component::Client::HTTP->spawn(Alias     => 'user_agent',
                                    Streaming => 1024);

POE::Component::Server::TCP->new
    ( Alias        => 'web_server',
      Port         => 8000,
      ClientFilter => 'POE::Filter::HTTPD',
      ClientInput  => \&client_input,
      InlineStates => { proxy_response => \&proxy_response }
    );

$poe_kernel->run();

sub client_input {
    my ($kernel, $heap, $request) = @_[KERNEL, HEAP, ARG0];

    # if request is actually response it just means that we've got an
    # error
    if($request->isa('HTTP::Response')) {
        $heap->{client}->put($request);
        $kernel->yield('shutdown');
        return;
    }

    $request->header(Connection => 'close');
    $kernel->post(user_agent => request => proxy_response => $request)
+;
}

sub proxy_response {
    my ($kernel, $heap, $request_packet, $response_packet)
        = @_[KERNEL, HEAP, ARG0, ARG1];

    my $request           = $request_packet->[0];
    my ($response, $data) = @$response_packet;

    unless($heap->{got_headers}) {
        $response->header(Connection => 'close');
        $heap->{client}->put($response);
        $heap->{got_headers}++;
    }

    if(defined $data) {
        $heap->{client}->set_output_filter(POE::Filter::Stream->new())
+;
        $heap->{client}->put($data);
    } else {
        $kernel->yield("shutdown");
    }
}

Note that at the time of writing it requires patched POE::Component::Client::HTTP.

--
Ilya Martynov, [e-mail]
CTO IPonWEB (UK) Ltd
Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
Personal website - http://martynov.org

[traveler]

Not perl, but a famous solution is dansguardian. It is GPL.

HTH, --traveler