Re: Do I need to use Coro instead of threads/forks

[Corion]

My advice is to not deal with Coro::cede but to use a Coro-aware LWP::UserAgent implementation, for example Coro::LWP. Then you can just use async { ... } to launch your processing in a non-blocking fashion.

I don't understand why you want to use Coro (or forks::BerkeleyDB) - maybe you want to communicate the data you fetch via a database or files instead. That would make the communication much easier, see perlipc.

[mohan2monks]

Thanks for your help corion
I am using SOAP::Lite to make calls to vendor api's which is internally using LWP.
I tried to override that but maybe done something wrong, if use Coro::LWP will it over ride that behavior, i went looking into source of LWP and it requires LWP::Useragent and stuff.


Your other suggestion, if i understand correctly is to fetch data store it in DB or files and read from there to serve as json.
The example that i gave of product data.
Basically this script is used to search inventory based on user search and results must be fetched from api's directly every time by my script add some of our data and return to user.
As this is inventory like that of airline seats,hotel rooms etc which is frequently changing. I cannot make cache of this information (cache misses are higher than cache hits used CHI driver BerkeleyDB) further the search combinations could be huge to fetch and write to data.
Biggest problem is main threads must wait till all threads return data, hence cannot use detach, detached thread writes data to file/db, main thread polls for write etc.

Any other approach i can follow?

[BrowserUk]

Biggest problem is main threads must wait till all threads return data,

Why is that a problem?

As I understand it, you have a cgi that accepts some user search terms. Once those search terms are returned to you, you then want to forward those terms to several vendor sites, aggregate the information they return to you, and then present the aggregation back to the user. (Is that correct? )

You cannot present the aggregation until you have all the data; so why is it a problem to wait for the threads to complete?

Unless you are hoping to present the data back to the user piecemeal, as you receive it?

In which case: set up a queue; detach the threads and have them post the data they receive to that queue. Then you main thread does not have to wait for all the threads to complete; it simply monitors the queue and deals with the data as it arrives.

[mohan2monks]

Yes BrowserUK your understanding is correct.
What i meant with main program must wait was exactly in the sense that i must aggregate content and hence cannot detach threads. (i never thought of queuing thinking it will be hard to manage)
Or can i try one of your excellent suggestion here Re: Why Coro?. using OS threads for each vendor and then use coros inside them to fulfill new requirement.
May be that should give me best of both.
Please correct me if i am wrong.

[Corion]

From what I understand of your problem, using Coro should work. Basically, your program should be something like your threads example, except that you use Coro::LWP and you use Coro instead of OS threads.

What have you tried so far that gives you problems?

Maybe you want to start with a small example program that does not talk to your vendor but tries to make several parallel requests to Google?

[mohan2monks]

I have created a test where i removed vendors and put in Coros instead of threads both in main and worker threads.
The script works fine even on actual data (tried with only two vendors i.e. 2 main threads and thread 1 opening 2 more inside it).
I am observing time taken to complete output and have not seen any improvement so far.
Further testing needs to be done to see if it is fetching data in parallel.
In 3 iterations it took 12 to 17 sec to finish.
With OS threads it should be around 10 to 12 sec. (need to confirm)