Submit job to Queue

[Anonymous Monk]

I have a perl program that runs an external process. The process is a circuit simulator. What I want to do is have multiple jobs put into a queue, and when the process completes, it runs the next file in the queue. This is step- by-step what I want to do:

1. Submit a system() call if that process is not running.
2. if process is already running, put system call in queue.
3. when 1st system call completes, grab next in queue, and run.

Thanks for the help!

[blakem]

So, you want to run multiple children in parallel, but be able to control how many are running at any one time (based on some sort of queue structure). This sounds like a job for Parallel::ForkManager.

-Blake

[Anonymous Monk]

I also forgot to mention that multiple people will be submitting these jobs from different instances of the same Perl program. So I think I'll need some text file holding the queue.

[blakem]

That shouldn't be much of a problem. It divides up nicely into the classic "consumers" and "producers" scenerio. Producers push stuff onto the queue, consumers take things off of it

First you have several producers (namely people submitting jobs) They dump job requests into a file, making sure they dont step on each others toes using file flocking.

Then you have the consumers... they consume the job requests. You can either have a single process read the job file and manage a bunch of subprocesses, or have each consumer process deal with fetching its own requests. I'd probably opt for having one manager process that takes care of reading the job queue and handing the requests off to the child processes. This is where Parallel::ForkManager comes in handy.

So you really have two programs... one used to submit the job request to the queue, and another to read the list and manage the children that will run the requests in parallel. Does that make sense?

-Blake

[tilly]

If you are on Linux or some form of Unix, install the at command. Scheduling jobs "at now" puts them into a queue, and if you want you can even give the queues names to have them go into different queues. Bonus: Configure at correctly and anything the job prints is sent as an email.

If you need something better, I would look search for job control solutions. If you don't find one that suits, you can always use a control table in a relational database for job control. That is more work, but you can allow jobs to be distributed across multiple machines in different locations.

[monk2b]

What I think you are looking for is a portable batch program.
There is a gpl one named portable batch system and a commercial
version named lsf
learning too
monk2b