SOAP::LITE client timeout makes ALL my Catalyst app to wait

[miguelele]

Hi!, in my Catalyst app, I have a very important connection to a remote server using SOAP with WSDL.

Everything works fine, but If the remote server goes down, ALL my app waits until the timeout expires. EVERYTHING, ALL the controllers and processes!!

If I set a 15 secs timeout, everything wait for 15 secs!! Any page from any user or connection can't be displayed during the timeout wait.

We have seen it just before going into production, when we have set the timeout to a higher value to prevent overloads in the remote server

I am very, very, very worried

Can you point me to somewhere to help with this?

sub check_result {
    my ($self, $code, $IP, $PORT) = @_;

    my $soap = SOAP::Lite->new( proxy => "http://$IP:$PORT/REMOTE_SOAP
+");

    $soap->autotype(0);
    $soap->default_ns('http://REMOTENAMESPACE/namespace/default');
    $soap->transport->timeout(15);
    
     $soap-> on_fault(sub { my($soap, $res) = @_; 
        eval { die ref $res ? $res->faultstring : $soap->transport->st
+atus };
          return ref $res ? $res : new SOAP::SOM;
       });
       
     my $som = $soap->call("remote_function",
         SOAP::Data->name( 'Entry1' )->value( $code ),
     );
 
    return $som->paramsout;
}

[trwww]

Without more information, my guess is you're using the script/myapp_server.pl and you're running it in single threaded mode.

Use the -fork option to script/myapp_server.pl so that the server can serve more than one request at a time, or more preferentially use a forking web server like mod_perl or something similar.

and/or

Create a job queue for the functionality that happens in check_result and then somehow poll the server for job completion (perhaps using AJAX).

[miguelele]

Thank you. I am using Nginx / Perl-fcgi, listening trough a port. (Mostly configured with default option...).

But the SOAP requests go trough another port, so I do not understand how it can hang all i/o and processes.

Migue.

[Khen1950fx]

You could try a different approach such as:

my $soap = SOAP::Lite->uri($uri)->proxy($proxyURL);

In your script:

#!/usr/bin/perl -l

use strict;
use warnings;
use SOAP::Lite;

my $uri = 'http://www.example.com';

sub check_result {
    my ( $self, $code, $IP, $PORT ) = @_;
    my $proxyURL = "http://$IP:$PORT/REMOTE_SOAP";

    my $soap = SOAP::Lite->uri($uri)->proxy($proxyURL);
    $soap->autotype(0);
    $soap->default_ns('http://REMOTENAMESPACE/namespace/default');
    $soap->proxy->timeout(15);
 
    $soap->on_fault(
        sub {
            my ( $soap, $res ) = @_;
            eval { die ref $res ? $res->faultstring : $soap->transport
+->status };
            return ref $res ? $res : new SOAP::SOM;
        }
    );

    my $som =
      $soap->call( "remote_function", SOAP::Data->name('Entry1')->valu
+e($code),
      );

    return $som->paramsout;
}

[miguelele]

Thank you, I tried your approach, but the situation remains the same

I also have started several FastCGI processes in Nginx, so when one process delays waiting for an answer, another one can be used by another user.

As I can see, the problem is not in the error handle, but in the time waiting for a response. Even if it is a Syncronous answer, I do not understand the it stops ALL the application.

Any other suggestion?, or a working example for a similar function?

Miguel

[shantanu_bhadoria]

use --workers option in plackup to add more than one workers when you run your server. preferable use starman instead. starman --preload-app --workers 10 /abc/xyz/myapp.psgi

[shantanu_bhadoria]

Btw if you are using Catalyst::Model::Adaptor in your process locking model, try replacing it with Catalyst::Model::Factory.

[Anonymous Monk]

trwww and other already suggested something along those lines, it is a good tip, but it won't change the bottleneck

[miguelele]

Hi, again. Last days I have been looking for answers from the point of view of the server (nginx), instead of a programming solution, because I found that I can avoid the app collapse starting more fastcgi processes. But the trick only runs until all the slots are filled.

Now I have read here and there some references telling that SOAP::LITE is single threaded.

Is it true??? Does it means that ALL my app, with ALL the visitors can only use one SOAP connection????

It is hard to believe. Please, can someone point me in the rigth direction?

Miguel

[Anonymous Monk]

The solution is obvious, stop trying to contact some remote SOAP server for EVERYTHING

[miguelele]

If yours is a serious response, it seems that I failed to describe my problem:

The app isnīt just a remote services eater. It does a lot of things: database querying, file processing, data management, sessions, authentication, etc. But there are some points where data MUST come from a remote WSDL service, and these points have become dangerous bottlenecks, because when that remote server goes down o delays its response, ALL THE APPLICATION waits until that process reach timeout or serves the response.

If yours is a funny response, please start your own thread, prefixing it with a "JOKE" keyword, so we can choose to read it when bored.

I have also crossposted this problem in Stack Overflow and SOAP::LITE list, whith any luck.

Any help will be really appreciated:
Miguel

[Anonymous Monk]

But there are some points where data MUST come from a remote WSDL service, and these points have become dangerous bottlenecks, because when that remote server goes down o delays its response, ALL THE APPLICATION waits until that process reach timeout or serves the response.

But you, yourself, said, it tries to contact remote remote SOAP server before every function/runmode -- a self imposed bottleneck.

There is no setting or trick or anything to change inside of SOAP::Lite or Catalyst to fix this problem.

You must fix it in your application.

See Watching long processes through CGI (Aug 02)

Maybe it sounds too simple, but no, I'm not trying to be funny (: its me again Catalyst (app_)server.pl hangs a lot:)

And now for the joke A tiger broke his

guitar but hes playing a new instrument now -- its a ukulele miguelele

:) Is it funny? Don't answer that, do you notice the difference?

 

Think of it a different way, instead of remote soap server, what would you do if it was local database (mysql) server?

Either your app waits and makes your user waits, or it quickly informs the user "here is some cached data" or "sorry, busy now, refresh me in a minute.

I don't have a link handy, but I remember reading an article (about reddit?) where they learned users hate to wait more than they hate old data or an error message, so cache lots of data.

Also worth reading are The Architecture of Open Source Applications (Volume 2): Scalable Web Architecture and Distributed Systems and mod_perl: Building a Large-Scale E-commerce site with Apache and mod_perl