Submit jobs¶
Decorate a function with a registered destination:
from molq import JobSpec, submit
local = submit("local", project_root="/absolute/project")
@local
def hello(name):
job = yield JobSpec.argv(["echo", f"hello {name}"])
return job
job = hello("molq")
Every yielded request is a JobSpec. Use JobSpec.argv(...) for one command;
use the full constructor for resources, scheduling, and execution plans.
Each yield receives its Job. The generator's return is the decorated call's
result. See the Python API for synchronous and async examples.
Arguments remain literal. Request shell interpretation explicitly with
ExecutionUnit.shell() when you need pipes or redirection.
Request an allocation¶
On a registered Slurm destination, request resources and queue settings separately from the command:
import json
from molq import Execution, ExecutionUnit, JobSpec, Resources, Scheduling, submit
spec = JobSpec(
resources=Resources(
nodes=1,
tasks=1,
cpus_per_task=4,
memory_bytes="8000000000",
time_limit_seconds=600,
),
scheduling=Scheduling(name="simulation", account="your-project"),
execution=Execution(
cwd="/scratch/your-user/project",
env={"OMP_NUM_THREADS": "4"},
units=(ExecutionUnit.argv("main", ["python", "simulate.py"]),),
),
)
hpc = submit("hpc", project_root="/shared/project")
print(json.dumps(hpc.preview(spec), ensure_ascii=False))
@hpc
def simulate():
job = yield spec
return job
memory_bytes is a decimal string of bytes per node. The scheduler rounds
up to its native unit. CPU allocation does not set your application's thread
count; specify that in its arguments or environment.
validate(spec) checks the request. preview(spec) also returns the rendered
script. Neither proves that an account, partition, or resource is available.
Execution plans¶
Run two branches together, then a final command. This example fits a single-node Slurm allocation with two CPUs:
import json
from molq import Parallel, Sequence, UnitRef
plan_spec = JobSpec(
resources=Resources(nodes=1, tasks=2, cpus_per_task=1),
execution=Execution(
units=(
ExecutionUnit.argv(
"a", ["echo", "a"], placement={"tasks": 1, "cpus_per_task": 1}
),
ExecutionUnit.argv(
"b", ["echo", "b"], placement={"tasks": 1, "cpus_per_task": 1}
),
ExecutionUnit.argv(
"finish", ["echo", "done"], placement={"tasks": 1, "cpus_per_task": 1}
),
),
plan=Sequence((Parallel((UnitRef("a"), UnitRef("b"))), UnitRef("finish"))),
),
)
print(json.dumps(hpc.preview(plan_spec), ensure_ascii=False))
Each unit appears once in the plan. A single unit needs no explicit plan.
Sequence stops on failure. Parallel waits for all started branches and
reports the first failure in plan order. Multiple units in a batch job require
an explicit task budget; parallel branches must fit that budget.
MPI¶
Declare ranks and threads, and configure the cluster's launcher:
from molq import Launch
mpi_spec = JobSpec(
resources=Resources(nodes=2, tasks=8, cpus_per_task=2),
execution=Execution(
units=(
ExecutionUnit.argv(
"main",
["./solver"],
launch=Launch(kind="mpi", ranks=8, threads=2),
),
)
),
)
On Slurm with launcher="srun", this renders an eight-rank native step.
The target must provide the executable and MPI environment.
Dependencies¶
A dependency refers to another allocation in the same cluster:
# previous is a submitted Job; next_execution is an Execution value.
dependent = JobSpec(
execution=next_execution,
scheduling=Scheduling(
dependencies=({"ref": previous.ref, "condition": "after_success"},)
),
)
Conditions are after, after_success, after_failure, and after_started.
The scheduler must support the requested condition exactly; see mappings.
Save job.ref after submission, or collection.refs for an array. Provide a
request_key to make one submission recoverable across serve restarts. Repeat
the same request and key to retrieve its receipt or reconcile with native
evidence. An unresolved submission returns OUTCOME_UNKNOWN and repeating
the key never resends it. Different parameters with the same key conflict.
Cancel keys remain scoped to one serve instance.
Slurm job arrays¶
Set Scheduling(array=JobArray((0, 1, 4), max_concurrent=2)) on a job spec.
Slurm receives one submission and runs each index in its own working directory.
The payload can read SLURM_ARRAY_TASK_ID. Python and TypeScript return a
JobCollection; CLI and RPC return array_id, refs, and snapshots.
Each element reference can be used independently for status, logs, dependencies,
and cancellation. The supported request size is 1–64 elements. Slurm decides
which indices and concurrency limits the site accepts.
For a Slurm plan with parallel branches, request one node, explicit tasks and
memory, and CPU-only resources. Each step receives its own CPU allocation
and a share of the requested memory proportional to its CPU count. Parallel
MPI uses the cluster's explicit srun launcher.
Native Slurm requeue¶
Set Scheduling(requeue=True) to opt into the site's native requeue policy.
Use job.requeue() or molq job requeue --ref @ref.json for an explicit
request on an incomplete Slurm job or individual array element. The native
scheduler decides whether to accept it. Acceptance does not confirm that the
job has restarted. Slurm may reset its submission time; use the new JobRef
from job list or native history for the restarted incarnation.
To rerun a finished array element, submit a new array with just the intended
indices and a fresh request key.
Slurm snapshots retain the raw state and native diagnostic fields. An allocation
reported as FAILED can have an out_of_memory reason when native accounting
shows an OOM step. The allocation's original state and the step evidence are
both preserved.
Project logs and outputs¶
Pass the existing target project directory as project_root when submitting.
Default logs and Shell receipts follow that project. Use an
artifact manifest to download
named user outputs. See files and logs for log access.