Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Grid Engine

by tsee (Curate)
on May 18, 2009 at 17:23 UTC ( #764681=note: print w/ replies, xml ) Need Help??


in reply to Grid Engine

We're running a ~200 CPU cluster using GridEngine and are quite satisfied. The queue master simply needs a decent amount of RAM and it can currently handle around 100k queued jobs with 4GB of RAM. Our jobs are typically adjusted to run about 1-12 hours, so any queuing overhead is negligible.

Now, I'm not aware of a full queuing system that's written in Perl and that beats Grid Engine on this scale. What I can point you at is a CLI tool for managing a Grid Engine installation. Well, not quite. It's not an admin's replacement for qmon: It's mostly useful for users who try to keep track of their jobs, put them on hold, clear their error state, add dependencies and so on.

However, I have to admit I wrote the aforementioned tool, so I'm biased.

In the end, the choice of tools really come down to the scale at which you're running this. If logging in to the nodes manually is still an option, then maybe Grid Engine isn't what you want. If you have very, very short jobs, it's certainly not what you want. Searching CPAN, I came up with these related modules: GRID::Machine, SSH::Batch. I've seen some others like TheSchwartz, but again, I don't know what exactly fits your usage.

Cheers,
Steffen


Comment on Re: Grid Engine
Re^2: Grid Engine
by Anonymous Monk on May 18, 2009 at 21:41 UTC
    Yes, I also administer an SGE cluster, and it works fine.

    I read the original question above a few times and there's not enough information to have a starting point.

    How large are the tasks you're scheduling?

    How many tasks per day?

    What do you mean the scheduling is not fair enough?

    I have a feeling you simply haven't read the 3 manuals available for sge, or don't know what to expect.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://764681]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2014-07-13 08:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (248 votes), past polls