Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

How can I convert my script for submitting SLURM jobs from Bash to Perl?

by erhan (Novice)
on Feb 18, 2015 at 20:18 UTC ( #1117141=perlquestion: print w/replies, xml ) Need Help??
erhan has asked for the wisdom of the Perl Monks concerning the following question:

I have the following Bash script for job submission to SLURM on a cluster:
#!/bin/bash #SBATCH -A 1234 #SBATCH -t 2-00:00 #SBATCH -n 24 module add gromacs srun resp.com
The #SBATCH lines are SLURM commands: #SBATCH -A 1234 is the project number (1234), #SBATCH -t 2-00:00 is the job time, #SBATCH -n 24 is the number of cores.  module add gromacs loads the Environment Module gromacs (gaussian is a computational quantum-chemistry program). srun is the SLURM command to launch a job. resp.com includes commands for gaussian and atom coordinates. I tried converting the Bash script to the following Perl script, but it didn't work. How can I do this in Perl?
#!/usr/bin/perl use strict; use warnings; use diagnostics; system ("#SBATCH -A 1234"); system ("#SBATCH -t 2-00:00"); system ("#SBATCH -n 24"); system ("module add xxx"); system ("srun resp.com ");

Replies are listed 'Best First'.
Re: How can I convert my script for submitting SLURM jobs from Bash to Perl?
by MidLifeXis (Monsignor) on Feb 18, 2015 at 20:54 UTC

    To add to AppleFritter's comments:

    • Each system call will have its own environment. If this is the akin to the module system that I used in the early '90s (setting up the environment and tools for a shell in a package-type manner), you will lose all of the module setup at the end of the system call.
    • Unless given some other reason to convert it to Perl, I would echo the suggestion to leave it in shell. This is what it is made for.

    --MidLifeXis

Re: How can I convert my script for submitting SLURM jobs from Bash to Perl?
by AppleFritter (Vicar) on Feb 18, 2015 at 20:26 UTC

    First of all, welcome to the Monastery!

    Second, a question: why do you want to convert this to Perl? (Especially when all the Perl script does is run shell commands.)

    Third, an observation: lines starting with a kitchen sink hash (#) are comments in shell scripts. Unless SLURM itself reads the shell script and interprets those lines, they won't do anything -- and if SLURM does that (which I sorta doubt, though admittedly I don't know a thing about it), then converting the shell script to a Perl script may be a vain endeavor to begin with.

      I want to convert it to Perl it because I will add on it the other parts after it works. It will be a long code. And I will be use it for some atomistic calculations.
Re: How can I convert my script for submitting SLURM jobs from Bash to Perl?
by Anonymous Monk on Feb 18, 2015 at 21:54 UTC

    A quick google brings up this, is this the "sbatch" / SLURM you're talking about? https://computing.llnl.gov/linux/slurm/sbatch.html

    If that's the case, and if my understanding from a quick skim is correct, then this isn't a plain bash script! There seems to be some preprocessing being done that reads the #SBATCH lines - as has already been noted, lines beginning with # are just comments to bash.

    You'll either need to find a monk who's worked with this system before, or give everyone else a quick overview of how this system works.

    I will still venture two guesses as to how this might still work with Perl. First, perhaps the system understands more than bash scripts. # lines are just comments to Perl too, so maybe this could work?

    #!/usr/bin/perl #SBATCH -A 1234 #SBATCH -t 2-00:00 #SBATCH -n 24 use warnings; use strict; system("module add gromacs")==0 or warn "Warning: module add failed, code $?"; system("srun resp.com")==0 or warn "Warning: srun failed, code $?";

    (This requires "module" and "srun" to be actual commands in your PATH, and not some directives that only sbatch understands.)

    Or, even though this isn't exactly a "conversion to Perl", it still gives some control to Perl - this uses IPC::Run3 to feed the "script" to sbatch via STDIN.

    #!/usr/bin/perl use warnings; use strict; use IPC::Run3 'run3'; my $script = <<'END_SCRIPT'; #!/bin/bash #SBATCH -A 1234 #SBATCH -t 2-00:00 #SBATCH -n 24 module add gromacs srun resp.com END_SCRIPT run3 ["sbatch"], \$script;

    Even though that's really just a simple wrapper, it still gives you some of the power of Perl, such as being able to generate the script dynamically or do advanced things with STDOUT and STDERR.

    I want to convert it to Perl it because I will add on it the other parts after it works.

    Sorry, I don't understand, could you explain a bit more?

      Thanks for your reply. Yes. I am talking about SLURM. I tried your first code (completely perl ones). It gives the following error: Can't exec "gaussian": No such file or directory at.... resp.com is only one job for run on cluster. I have about 1000 jobs. So I have to convert it to perl script.

        I think the first script needs to be:

        #!/usr/bin/perl #SBATCH -A 1234 #SBATCH -t 2-00:00 #SBATCH -n 24 use warnings; use strict; system("module add gromacs && srun resp.com")==0 or warn "Warning: command failed, code $?";

        module add gromacs is likely modifying the "local" runtime environment as MidLifeXis explained in his post, above.

        Sorry, but I don't see how the script shown can produce that error message; perhaps you modified it, in that case you need to post the full code, or, the error message is actually coming from one of the two executed commands, in which case that needs to be debugged...

        I'm not sure how many monks here might already have experience with SLURM, gromacs, et al, but if it's only a few, the rest of us will need all the information you can provide so that we can help you best. To that effect, it would be good if you could read I know what I mean. Why don't you? and How do I post a question effectively?

Re: How can I convert my script for submitting SLURM jobs from Bash to Perl?
by Anonymous Monk on Feb 18, 2015 at 23:12 UTC

    There seems to be an example of how to start a bunch of jobs in series or parallel using Perl here: http://www.zedat.fu-berlin.de/HPC/EN/SorobanQueueingSystem#MultistepSerialJob

    Also there appears to be a Perl API for Slurm: https://github.com/SchedMD/slurm/tree/master/contribs/perlapi - easier to read documentation appears to be here: http://manpages.ubuntu.com/manpages/trusty/man3/Slurm.3pm.html

    (Disclaimer: I have no idea about SLURM, these are just some Google results that appear relevant.)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1117141]
Approved by AppleFritter
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2018-11-21 04:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My code is most likely broken because:
















    Results (237 votes). Check out past polls.

    Notices?