The model that I like to use for big, long-running activities is one that has been around forever: batch jobs. x jobs are running simultaneously within a management framework that I didnít write, and the remaining y are queued.
Yes, after analyzing all advantages and disadvantages of process and threads, I also felt individual process is best option...it can not take the risk of failing all jobs if any one job fails, (even it has to avoid single failure) )...huge data is leaded for main process, whcih will not be ideal scenario to start threads....and batch is simple, tested concept whcih works, so on my way of getting this done using process.
Choose your language-of-choice in which to implement them.
If it had been from scratch, I would have thought of this option, but in this case everything is written in Perl and written well (except multiprocessing part) working reliably from past 6 years (with agile mode updates).