Re: Grid Engine

[tsee]

We're running a ~200 CPU cluster using GridEngine and are quite satisfied. The queue master simply needs a decent amount of RAM and it can currently handle around 100k queued jobs with 4GB of RAM. Our jobs are typically adjusted to run about 1-12 hours, so any queuing overhead is negligible.

Now, I'm not aware of a full queuing system that's written in Perl and that beats Grid Engine on this scale. What I can point you at is a CLI tool for managing a Grid Engine installation. Well, not quite. It's not an admin's replacement for qmon: It's mostly useful for users who try to keep track of their jobs, put them on hold, clear their error state, add dependencies and so on.

However, I have to admit I wrote the aforementioned tool, so I'm biased.

In the end, the choice of tools really come down to the scale at which you're running this. If logging in to the nodes manually is still an option, then maybe Grid Engine isn't what you want. If you have very, very short jobs, it's certainly not what you want. Searching CPAN, I came up with these related modules: GRID::Machine, SSH::Batch. I've seen some others like TheSchwartz, but again, I don't know what exactly fits your usage.

Cheers,
Steffen

[Anonymous Monk]

Yes, I also administer an SGE cluster, and it works fine.

I read the original question above a few times and there's not enough information to have a starting point.

How large are the tasks you're scheduling?

How many tasks per day?

What do you mean the scheduling is not fair enough?

I have a feeling you simply haven't read the 3 manuals available for sge, or don't know what to expect.