in reply to Actively monitor space
There are many job-monitor systems out there for Unix and Linux, and one of the things they can regulate is “expected disk-space usage.” But they really can’t observe how much disk space any particular job is using, nor can they necessarily regulate it (even with quota). You need to know the jobs.
Many shops try to do job scheduling with crontabs, and they get into trouble because you really can’t predict the completion-time of one unit of work. You wind up either wasting time or doubling-up on resources.
So-called “workflow monitoring” systems can be surprisingly effective. The various components of what we call “jobs” can in fact all run as child-processes of a parent that is simply forking children and waiting for them to complete. Heck, if the workflow is predictable and unchanging, that can be how you run jobs with a great deal of control, e.g. driven by a Perl script. All the work is done under the auspices of one parent pid.