SLURM partition not detected from config

[gcapes]

Trying to run this simple workflow on CSF3 using slurm,

template_components:
  task_schemas:
  - objective: locate_matlab
    actions:
    - commands:
      - command: which matlab
      environments:
      - scope:
          type: any
        environment: matlab_env  

tasks:
- schema: locate_matlab

resources:
  any:
    scheduler_args:
      options:
        --time: 00:30:00

I came up against this error:

sbatch: error: Batch job submission failed: Invalid partition name specified
Traceback (most recent call last):
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/bin/matflow", line 8, in <module>
    sys.exit(cli())
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/hpcflow/sdk/cli.py", line 253, in make_and_submit_workflow
    out = app.make_and_submit_workflow(
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/hpcflow/sdk/app.py", line 1644, in <lambda>
    return lambda *args, **kwargs: func(*args, **kwargs)
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/hpcflow/sdk/app.py", line 2897, in _make_and_submit_workflow
    submitted_js = wk.submit(
  File "/net/scratch/mbexegc2/matflow-demo-workflows/.venv/lib64/python3.9/site-packages/hpcflow/sdk/core/workflow.py", line 3336, in submit
    raise WorkflowSubmissionFailure(exceptions)
hpcflow.sdk.core.errors.WorkflowSubmissionFailure: 
Some jobscripts in submission index 0 could not be submitted:
Jobscript 0 at path: '/net/scratch/mbexegc2/matflow-demo-workflows/wheresmatlab_2025-04-17_103905/artifacts/submissions/0/jobscripts/js_0.sh'
Submit command: ['sbatch', '--parsable', '/net/scratch/mbexegc2/matflow-demo-workflows/wheresmatlab_2025-04-17_103905/artifacts/submissions/0/jobscripts/js_0.sh'].
Reason: 'Non-empty stderr from submit command.'
Submission stderr:
  sbatch: error: Batch job submission failed: Invalid partition name specified

This was side-stepped by specifying the partition in the jobscript as shown below:

template_components:
  task_schemas:
  - objective: locate_matlab
    actions:
    - commands:
      - command: which matlab
      environments:
      - scope:
          type: any
        environment: matlab_env  

tasks:
- schema: locate_matlab

resources:
  any:
    scheduler_args:
      options:
        --time: 00:30:00
        --partition: serial

[gcapes]

This is my config

╭────────────────────────────────────────────────────────────────────────────────────────────────────── Config 'manchester-CSF3-slurm' ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│  machine               'manchester-CSF3-new'                                                                                                                                                                                               │
│  log_file_path         PosixPath('/mnt/iusers01/support/mbexegc2/.matflow-new/logs/MatFlow_v0.3.0a155.log')                                                                                                                                │
│  task_schema_sources   []                                                                                                                                                                                                                  │
│  parameter_sources     []                                                                                                                                                                                                                  │
│  command_file_sources  []                                                                                                                                                                                                                  │
│  environment_sources   [PosixPath('/mnt/iusers01/support/mbexegc2/.matflow-new/envs-slurm.yaml')]                                                                                                                                          │
│  default_scheduler     'slurm'                                                                                                                                                                                                             │
│  default_shell         'bash'                                                                                                                                                                                                              │
│  schedulers            {                                                                                                                                                                                                                   │
│                            'direct': {'defaults': {}},                                                                                                                                                                                     │
│                            'slurm': {                                                                                                                                                                                                      │
│                                'defaults': {'shebang_args': '--login'},                                                                                                                                                                    │
│                                'partitions': {                                                                                                                                                                                             │
│                                    'serial': {'num_cores': [1, 1, 1]},                                                                                                                                                                     │
│                                    'multicore': {                                                                                                                                                                                          │
│                                        'num_nodes': [1, 1, 1],                                                                                                                                                                             │
│                                        'num_cores': [2, 1, 168],                                                                                                                                                                           │
│                                        'num_cores_per_node': [2, 1, 168],                                                                                                                                                                  │
│                                        'parallel_modes': ['distributed', 'shared', 'hybrid']                                                                                                                                               │
│                                    },                                                                                                                                                                                                      │
│                                    'multinode': {'num_nodes': [2, 1, None], 'num_cores': [80, 40, None], 'num_cores_per_node': [40, 40, 40], 'parallel_modes': ['distributed', 'hybrid']}                                                  │
│                                }                                                                                                                                                                                                           │
│                            }                                                                                                                                                                                                               │
│                        }                                                                                                                                                                                                                   │
│  shells                {'bash': {'defaults': {}}}                                                                                                                                                                                          │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[gcapes]

Adam is aware of this bug, and we have a workaround for now.

[dkfellows]

Hmm, it's supposed to pick the first compatible partition... except it's probably saying "I don't know how many cores are wanted for this job, so that partition can't be matched against as that has a range of cores set".

I'm guessing that saying that the job in question only needs one core will make it work. If so, that's a superior workaround (as it's a portable fix) and we can then fix things by adjusting the resource request to say that the number of cores wanted is 1 if not already known (when doing this match-by-approximation algorithm in SlurmPosix.process_resources).

[dkfellows]

Linked PR says "If we're choosing partitions by matching resources, and the user hasn't actually stated what resources they want, assume they just want a single core." It's a bit of a janky hack of a fix, but it will at least mean that small jobs will run (assuming there's a "serial" partition in the first place). User controls which they get in part by the order of partition descriptions.

[gcapes]

I don't think that's the (only) issue here. I get the same error from this workflow (last two lines modified compared with above):

template_components:
  task_schemas:
  - objective: locate_matlab
    actions:
    - commands:
      - command: which matlab
      environments:
      - scope:
          type: any
        environment: matlab_env  

tasks:
- schema: locate_matlab

resources:
  any:
    scheduler_args:
      directives:
        --time: 00:30:00
#        --partition: serial
    num_cores: 1

[gcapes]

@dkfellows

[dkfellows]

Whoops; attention was elsewhere.

Adapted workflow to this test case:

template_components:
  task_schemas:
  - objective: locate_bash
    outputs:
      - parameter: bash_location
    actions:
    - commands:
      - command: which bash
        stdout: <<parameter:bash_location>>
      environments:
      - scope:
          type: any
        environment: null_env

tasks:
- schema: locate_bash

resources:
  any:
    scheduler_args:
      directives:
        --time: 00:30:00
#        --partition: serial
    num_cores: 1

@pytest.mark.slurm
def test_workflow_2(tmp_path: Path, null_config):
    hf.config.add_scheduler("slurm")
    wk = make_test_data_YAML_workflow("workflow_2_slurm.yaml", path=tmp_path)
    wk.submit(wait=True, add_to_known=False)
    bash_location = wk.tasks[0].elements[0].outputs.bash_location
    assert isinstance(bash_location, hf.ElementParameter)
    assert "bash" in bash_location.value