Re: Queuing system for running a perl program

[wjw]

So many questions are generated by a question like this.

For example, is the user informed that the "process is running", come back later for your answer, or "your results will be ready in about x seconds/minutes."?
Why does processing take so long?
etc...

If it were me, I would start by considering changing that PHP script to Perl, if for no other reason than to reduce the number of languages by one.
The queue system seems like a reasonable approach, as long as you can communicate it back to your user, so they are not left wondering if something is happening.
That, to my way of thinking, implies that Perl should do the queuing and, if required, inform PHP and thus the user.

Your back-end is already working, though slowly. I would also examine those 'stand-alone' software pieces and see if what they do can be done better(maybe by Perl).

Another question: are these jobs that are running somehow attached to a session?
How do you ensure the results come back to the user that submitted?
What about jobs where the user leaves before the results come back? Is that job going to be de-queued?

Hope these thoughts are helpful, and best of luck with your project.

[Anonymous Monk]

Hi guys,
thanks to all for your time...Some clarifications:
1) No, the "Extra" software that is being executed, cannot be re-written in Perl, there are complex programs, can't touch them...
2)What I am thinking to do is have a random-string generator subroutine, so that the files generated for each user are separated from those of the others. Sound reasonable approach?
Is there somehow a way to say, have Perl tell me that "Now I started processing this ID" and then associate this unique ID to the current process and Perl tell me that "Now I finished with this ID"? In that scenario, I would know when each full-process has ended and re-direct the user to the output page... Maybe some kind of intermediate "Your query is being processed page" could be helpful? In that case, is this done by PHP, Perl, Javacript?

[GrandFather]

You may be interested in Data::GUID for generating your unique process ID.

If your processing code runs on a single system you can use PIDs to uniquely identify currently running processes. If you are running across a number of systems use a GUID to identify a task and map that to a system name and PID to allow monitoring task progress. This stuff is commonly enough done that much of it is likely already provided in a module, but my quick CPAN search didn't turn up anything promising.

[wjw]

As mentioned by another poster(and a good point), there is in fact, a lot of 'prior art' out there.

Also, as mentioned by yet another poster, there are in fact existing schedulers which would handle this.

What you propose is certainly doable with Perl(and others). Again, I have to wonder how long the user will be waiting.
If the user is expected to wait a few seconds, no problem. If longer...? Regardless, I personally would have Perl put out
that "please wait for..." page. Your back-end knows what is going on, let it inform the front end. There is no penalty that I can
think of for doing it that way. To keep the user more or less engaged, have Perl spit out some javascript which probes the server
for the state of the job and updates the user every 10 seconds or so...

Just a thought... .

[hippo]

What I am thinking to do is have a random-string generator subroutine, so that the files generated for each user are separated from those of the others. Sound reasonable approach?

It may seem like that now, but don't do this. Sooner or later the generator will repeat one of the random strings and then the trouble begins. Better to use a truly unique identifier (which could be as simple as an incrementing integer).