Dancer as a proxy

[gsiems]

I'm trying to create a new route in dancer that will serve as a proxy of sorts to another service. This other service returns various kinds of files of varying size.

For small files, the following snippet works fine:

  my $ua = LWP::UserAgent->new(timeout => 180,);
  $ua->env_proxy;

  my $rsp = $ua->head($new_route);
  send_error('Bad Gateway (head)', 502) unless (ref $rsp && $rsp->is_s
+uccess);

  foreach my $key ('Content-Encoding', 'Content-Length', 'Content-Type
+', 'Last-Modified') {
      my $value = $rsp->header($key) || '';
      header($key => $value) if ($value);
  }

  $rsp = $ua->get($new_route);
  send_error('Bad Gateway (get)', 502) unless (ref $rsp && $rsp->is_su
+ccess);
  return $rsp->content;

The problem that I'm having is that, for large files, I would like to be able to stream the response. Now LWP::UserAgent allows for grabbing the response in chunks using a callback and Dancer allows sending data in chunks using send_file with a callback but I (as yet) have no clue how to tie the two together in any workable fashion.

Any ideas how to make this work?

[Corion]

Take a look at PSGI and what it says about streaming responses. I don't know if Dancer properly supports streaming, but only because I haven't looked at it closely enough.

[Anonymous Monk]

streaming support depends on the webserver as much as the framework , and a grep http://search.cpan.org/grep?cpanid=YANICK&release=Dancer-1.3113&string=streaming&i=1&n=1&C=3 shows Dancer higher than 1.3072 supports streaming

Streaming data using dancer, Re^3: Resume downloads, Re^2: file download with connection problems, http://search.cpan.org/perldoc/Dancer#send_file

This was really hard to write as the documentation has holes on this, there are no tests on this in test suite, and Dancer hides the errors from these callbacks, so you have to eval { ...; 1 } or warn $@;

get '/stream_example' => sub {
    return send_file(
        \'fake contents to be replaced in override',
        streaming => 1,
        callbacks => {
            override => sub {
eval {
    my ( $respond, $response ) = @_;
    require LWP; my $ua = LWP::UserAgent->new;
    my $writer ;
    $ua->get(
        'http://www.example.com',
        ':content_cb' => sub {
            my ( $data, $response, $protocol ) = @_;
            if(not $writer ){
                my $h = $response->headers ;        
                my $r = [
                    $response->code,
                    [
                        map { $_ => $h->{$_}, }
                        'last-modified' , 'date' , 'content-length' , 
+'etag' , 'content-type' 
                    ],
                ];
                $writer = $respond->( $r );
            }
            $writer->write( $data );
        }, ## end of :content_cb
    ); ## end of $ua->get
    
} or warn "\n\n OH NO\n$@\n\n";
                return;
            }, ## end of override
        },
    ); ## end of send_file
}; ## end of get '/stream_example'

[gsiems]

Thank you Anonymous Monk! I did need to change it a bit to make it work as I was having issues getting the headers information and things appear to working rather nicely now.

For future reference, and in case others find it useful, the modified send_file is:

    return send_file(
        \'fake contents to be replaced in override',
        streaming => 1,
        callbacks => {
            override => sub {
eval {
    my ( $respond, $response ) = @_;
    require LWP;
    my $ua = LWP::UserAgent->new;
    my $writer ;

    $ua->get(
        $new_route,
        ':content_cb' => sub {
            my ( $data, $response, $protocol ) = @_;

            if(not $writer ){
                my $h = $response->headers ;
                my %m = map {$_ => $h->header($_)} 
                    ('Content-Length', 'Content-Type', 'Last-Modified'
+);
                $writer = $respond->( [ $response->code, [ %m ] ] );
            }

            $writer->write( $data );
        }, ## end of :content_cb
    ); ## end of $ua->get
} or Dancer::Logger::error("stream-file: bad eval $@");

                return;
            }, ## end of override
        },
    ); ## end of send_file