WES 1.1.0

.dockerignore

@@ -5,5 +5,5 @@
 !MANIFEST.in
 !README.md
 !sapporo
-!sapporo-wes-1-0-1-openapi-spec.yml
+!sapporo-wes-1-1-0-openapi-spec.yml
 !setup.py

MANIFEST.in

@@ -1,5 +1,5 @@
 include LICENSE
-include sapporo-wes-1-0-1-openapi-spec.yml
+include sapporo-wes-1-1-0-openapi-spec.yml
 include sapporo/auth_config.json
 include sapporo/auth_config.schema.json
 include sapporo/executable_workflows.json

README.md

@@ -12,7 +12,7 @@
 The sapporo-service is a standard implementation conforming to the [Global Alliance for Genomics and Health](https://www.ga4gh.org) (GA4GH) [Workflow Execution Service](https://github.com/ga4gh/workflow-execution-service-schemas) (WES) API specification.
 
 We have also extended the API specification.
-For more details, please refer to [`./sapporo-wes-1-0-1-openapi-spec.yml`](./sapporo-wes-1-0-1-openapi-spec.yml) for more details.
+For more details, please refer to [`./sapporo-wes-1-1-0-openapi-spec.yml`](./sapporo-wes-1-1-0-openapi-spec.yml) for more details.
 
 One of the key features of the sapporo-service is its ability to abstract workflow engines, making it easy to adapt various workflow engines to the WES standard.
 Currently, we have verified compatibility with the following workflow engines:
@@ -86,7 +86,7 @@ optional arguments:
   --service-info        Specify the `service-info.json` file. The
                         `supported_wes_versions` and `system_state_counts` will
                         be overwritten by the application.
-  --executable-workflows 
+  --executable-workflows
                         Specify the `executable-workflows.json` file.
   --run-sh              Specify the `run.sh` file.
   --url-prefix          Specify the prefix of the URL (e.g., --url-prefix /foo
@@ -107,7 +107,6 @@ Note that startup arguments take precedence over environment variables.
 #### Standard WES Mode
 
 In this mode, the sapporo-service conforms to the standard WES API specification.
-However, it's important to note that when using the sapporo-service, there is a deviation from the standard WES API specification: **you are required to specify `workflow_engine_name` in the request parameter of `POST /runs`.** This is due to the sapporo-service's ability to abstract workflow engines, as mentioned above.
 
 #### Execute Only Registered Workflows Mode
 
@@ -189,13 +188,13 @@ You can override this location using the startup argument `--service-info` or th
 
 The sapporo-service allows you to generate download links for files and directories located under the `run_dir`.
 
-For more details, please refer to the `GetData` section in [`./sapporo-wes-1-0-1-openapi-spec.yml`](./sapporo-wes-1-0-1-openapi-spec.yml).
+For more details, please refer to the `GetData` section in [`./sapporo-wes-1-1-0-openapi-spec.yml`](./sapporo-wes-1-1-0-openapi-spec.yml).
 
 ### Parse Workflow
 
 The sapporo-service offers a feature to inspect the type, version, and inputs of a workflow document.
 
-For more details, please refer to the `ParseWorkflow` section in [`./sapporo-wes-1-0-1-openapi-spec.yml`](./sapporo-wes-1-0-1-openapi-spec.yml).
+For more details, please refer to the `ParseWorkflow` section in [`./sapporo-wes-1-1-0-openapi-spec.yml`](./sapporo-wes-1-1-0-openapi-spec.yml).
 
 ### Generate RO-Crate
 

sapporo-wes-1-0-1-openapi-spec.yml → sapporo-wes-1-1-0-openapi-spec.yml

@@ -3,7 +3,7 @@ info:
   title: >-
     GA4GH Workflow Execution Service API specification extended for
     the sapporo-service
-  version: sapporo-wes-1.0.1-oas3
+  version: sapporo-wes-1.1.0-oas3
   description: >-
     The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems and monitor their execution.
     This API lets users run a single workflow on multiple different platforms, clouds, and environments.
@@ -712,6 +712,7 @@ components:
         - SYSTEM_ERROR
         - CANCELED
         - CANCELING
+        - PREEMPTED
       default: UNKNOWN
       description: >-
         - UNKNOWN: The state of the task is unknown.
@@ -740,6 +741,10 @@ components:
         - CANCELED: The task was canceled by the user.
 
         - CANCELING: The task was canceled by the user and is in the process of stopping.
+
+        - PREEMPTED: The task is stopped (preempted) by the system. The reasons for this would be tied to
+        the specific system running the job. Generally, this means that the system reclaimed the compute
+        capacity for reallocation.
     RunListResponse:
       type: object
       additionalProperties: false
@@ -750,7 +755,9 @@ components:
           type: array
           additionalProperties: false
           items:
-            $ref: "#/components/schemas/RunStatus"
+            anyOf:
+              - $ref: "#/components/schemas/RunStatus"
+              - $ref: "#/components/schemas/RunSummary"
           description: >-
             A list of workflow runs that the service has executed or is executing.
             The list is filtered to only include runs that the caller has permission to see.
@@ -761,6 +768,9 @@ components:
             An empty string indicates there are no more items to return.
       description: >-
         The service will return a RunListResponse when receiving a successful RunListRequest.
+        DEPRECIATION WARNING: The use of `RunStatus` as the schema for `runs` array items will
+        not be permitted from the next major version of the specification (2.0.0) onwards. We
+        encourage implementers to use `RunSummary` instead.
     RunLog:
       type: object
       additionalProperties: false
@@ -798,7 +808,7 @@ components:
       type: object
       additionalProperties: false
       required:
-        - workflow_engine_name
+        - workflow_engine
       properties:
         workflow_params:
           type: string
@@ -826,10 +836,14 @@ components:
           format: application/json
           description: >-
             A key-value map of arbitrary metadata outside the scope of `workflow_params` but useful to track with this run request
-        workflow_engine_name:
+        workflow_engine:
           type: string
           description: >-
             Specify the name of the workflow engine to run a workflow.
+        workflow_engine_version:
+          type: string
+          description: >-
+            Specify the version of the workflow engine to run a workflow.
         workflow_engine_parameters:
           type: string
           format: application/json
@@ -881,7 +895,7 @@ components:
           type: string
           description: workflow run ID
     RunStatus:
-      type: object
+      type: object, returned by server during listing
       required:
         - run_id
         - state
@@ -890,7 +904,27 @@ components:
           type: string
         state:
           $ref: "#/components/schemas/State"
-      description: Small description of a workflow run, returned by server during listing
+      description: State information of a workflow run
+    RunSummary:
+        title: RunSummary,
+        allOf:
+          - $ref: "#/components/schemas/RunStatus"
+          - type: object
+            properties:
+              start_time:
+                type: string
+                description: When the run started executing, in ISO 8601 format "%Y-%m-%dT%H:%M:%SZ"
+              end_time:
+                type: string
+                description: When the run stopped executing (completed, failed, or cancelled), in ISO 8601 format "%Y-%m-%dT%H:%M:%SZ"
+              tags:
+                type: object,
+                description: Arbitrary key/value tags added by the client during run creation
+                additionalProperties:
+                  type: string
+            required:
+              - tags
+        description: Small description of a workflow run
     WorkflowTypeVersion:
       type: object
       additionalProperties: false

sapporo/model/factory.py

@@ -16,7 +16,7 @@ def generate_service_info() -> ServiceInfo:
     with current_app.config["SERVICE_INFO"].open(mode="r", encoding="utf-8") as f:
         service_info: ServiceInfo = json.load(f)
 
-    service_info["supported_wes_versions"] = ["sapporo-wes-1.0.1"]
+    service_info["supported_wes_versions"] = ["sapporo-wes-1.1.0"]
     service_info["system_state_counts"] = count_system_state()
     service_info["tags"]["get_runs"] = current_app.config["GET_RUNS"]
     service_info["tags"]["workflow_attachment"] = current_app.config["WORKFLOW_ATTACHMENT"]

sapporo/model/sapporo_wes_1_0_1.py

@@ -100,7 +100,7 @@ class RunRequest(TypedDict):
     workflow_type: Optional[WorkflowTypes]
     workflow_type_version: Optional[str]
     tags: Optional[str]
-    workflow_engine_name: str
+    workflow_engine: str
     workflow_engine_parameters: Optional[str]
     workflow_url: Optional[str]
     workflow_name: Optional[str]

sapporo/run.py

@@ -195,7 +195,7 @@ def convert_wf_engine_params_str(run_request: RunRequest) -> str:
         service_info = generate_service_info()
         default_wf_engine_dict = service_info["default_workflow_engine_parameters"]
         default_wf_engine_params = default_wf_engine_dict.get(
-            run_request["workflow_engine_name"], [])
+            run_request["workflow_engine"], [])
         for default_wf_engine_param in default_wf_engine_params:
             params.append(default_wf_engine_param.get("name", ""))
             params.append(default_wf_engine_param.get("default_value", ""))

sapporo/run.sh

@@ -6,8 +6,8 @@ function run_wf() {
   download_workflow_attachment
   echo "RUNNING" >${state}
   date +"%Y-%m-%dT%H:%M:%S" >${start_time}
-  # e.g. when wf_engine_name=cwltool, call function run_cwltool
-  local function_name="run_${wf_engine_name}"
+  # e.g. when wf_engine=cwltool, call function run_cwltool
+  local function_name="run_${wf_engine}"
   if [[ "$(type -t ${function_name})" == "function" ]]; then
     ${function_name}
     generate_outputs_list
@@ -127,7 +127,7 @@ function run_streamflow() {
 
 function cancel() {
   # Pre-cancellation procedures
-  if [[ ${wf_engine_name} == "cwltool" ]]; then
+  if [[ ${wf_engine} == "cwltool" ]]; then
     cancel_cwltool
   fi
   cancel_by_request
@@ -217,7 +217,7 @@ cmd="${run_dir}/cmd.txt"
 task_logs="${run_dir}/task.log"
 
 # Meta characters are escaped.
-wf_engine_name=$(jq -r ".workflow_engine_name" ${run_request})
+wf_engine=$(jq -r ".workflow_engine" ${run_request})
 wf_url=$(jq -r ".workflow_url" ${run_request})
 wf_engine_params=$(head -n 1 ${wf_engine_params_file})
 

sapporo/validator.py

@@ -51,10 +51,10 @@ def validate_run_request(run_id: str) -> RunRequest:
     tags = request.form.get("tags", None)
 
     # Fields: require validation, but not related to the registered mode
-    wf_engine_name = request.form.get("workflow_engine_name", None)
+    wf_engine = request.form.get("workflow_engine", None)
     wf_attachment_str = request.form.get("workflow_attachment", None)
     wf_attachment_files = request.files.getlist("workflow_attachment")
-    wf_engine_name = validate_wf_engine_name(wf_engine_name)
+    wf_engine = validate_wf_engine(wf_engine)
     wf_attachment = validate_wf_attachment(run_id, wf_attachment_str, wf_attachment_files)
 
     # Fields: require validation, related to the registered mode
@@ -95,23 +95,23 @@ def validate_run_request(run_id: str) -> RunRequest:
         "workflow_type": wf_type,
         "workflow_type_version": wf_type_version,
         "tags": tags,
-        "workflow_engine_name": wf_engine_name,
+        "workflow_engine": wf_engine,
         "workflow_engine_parameters": wf_engine_params,
         "workflow_url": wf_url,
         "workflow_name": wf_name,
         "workflow_attachment": json.dumps(wf_attachment),
     }
 
 
-def validate_wf_engine_name(wf_engine_name: Optional[str]) -> str:
-    if wf_engine_name is None:
-        abort(400, "Workflow engine name is required.")
+def validate_wf_engine(wf_engine: Optional[str]) -> str:
+    if wf_engine is None:
+        abort(400, "Workflow engine is required.")
     service_info = generate_service_info()
-    wf_engines_names = list(service_info["workflow_engine_versions"].keys())
-    if wf_engine_name not in wf_engines_names:
-        abort(400, f"Workflow engine name `{wf_engine_name}` is not supported.")
+    wf_engines = list(service_info["workflow_engine_versions"].keys())
+    if wf_engine not in wf_engines:
+        abort(400, f"Workflow engine `{wf_engine}` is not supported.")
 
-    return wf_engine_name
+    return wf_engine
 
 
 def validate_registered_only_mode() -> None:
@@ -212,14 +212,14 @@ def validate_meta_characters(_type: str, content: str) -> None:
     """\
     This function checks the validity of the string that will be evaluated in the 'eval'
     command within run.sh. The string could be of the type 'workflow_url',
-    'workflow_engine_name', or 'workflow_engine_params'. If any of these strings contain
+    'workflow_engine', or 'workflow_engine_params'. If any of these strings contain
     characters from the list of prohibited characters below, the operation will be aborted.
 
     This function is invoked as shown below in the POST /runs endpoint:
 
     validate_meta_characters("workflow_engine_params", joined_params)
     validate_meta_characters("workflow_url", run_request["workflow_url"])
-    validate_meta_characters("workflow_engine_name", run_request["workflow_engine_name"])
+    validate_meta_characters("workflow_engine", run_request["workflow_engine"])
     """
     prohibited_characters = [";", "!", "?", "(", ")", "[", "]", "{", "}", "*", "\\", "&", r"`", "^", "<", ">", "|", "$"]
     for char in content:

tests/curl_example/cromwell_bamstats_cwl.sh

@@ -24,7 +24,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=CWL" \
   -F "workflow_type_version=v1.0" \
   -F "workflow_url=./Dockstore.cwl" \
-  -F "workflow_engine_name=cromwell" \
+  -F "workflow_engine=cromwell" \
   -F "tags=${tags}" \
   -F "workflow_attachment=@${workflow}" \
   -F "workflow_attachment=@${data}" \

tests/curl_example/cromwell_bamstats_wdl.sh

@@ -24,7 +24,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=WDL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./Dockstore.wdl" \
-  -F "workflow_engine_name=cromwell" \
+  -F "workflow_engine=cromwell" \
   -F "tags=${tags}" \
   -F "workflow_attachment=@${workflow}" \
   -F "workflow_attachment=@${data}" \

tests/curl_example/cwltool_attach_all_files.sh

@@ -33,7 +33,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=CWL" \
   -F "workflow_type_version=v1.0" \
   -F "workflow_url=./trimming_and_qc.cwl" \
-  -F "workflow_engine_name=cwltool" \
+  -F "workflow_engine=cwltool" \
   -F "workflow_attachment=@${workflow}" \
   -F "workflow_attachment=@${tool_1}" \
   -F "workflow_attachment=@${tool_2}" \

tests/curl_example/cwltool_registered_workflow.sh

@@ -20,7 +20,7 @@ response=$(curl -fsSL -X POST \
   -H "Content-Type: multipart/form-data" \
   -F "workflow_name=Example workflow - CWL - Trimming and QC" \
   -F "workflow_params=${workflow_params}" \
-  -F "workflow_engine_name=cwltool" \
+  -F "workflow_engine=cwltool" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)
 
 if [[ $? -ne 0 ]]; then

tests/curl_example/cwltool_remote_workflow.sh

@@ -22,7 +22,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=CWL" \
   -F "workflow_type_version=v1.0" \
   -F "workflow_url=https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/trimming_and_qc_remote.cwl" \
-  -F "workflow_engine_name=cwltool" \
+  -F "workflow_engine=cwltool" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)
 
 if [[ $? -ne 0 ]]; then

tests/curl_example/cwltool_tags_workflow_name.sh

@@ -26,7 +26,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type_version=v1.0" \
   -F "workflow_url=https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/trimming_and_qc_remote.cwl" \
   -F "tags=${tags}" \
-  -F "workflow_engine_name=cwltool" \
+  -F "workflow_engine=cwltool" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)
 
 if [[ $? -ne 0 ]]; then

tests/curl_example/cwltool_workflow_engine_parameters.sh

@@ -25,7 +25,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=CWL" \
   -F "workflow_type_version=v1.0" \
   -F "workflow_url=https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/trimming_and_qc_remote.cwl" \
-  -F "workflow_engine_name=cwltool" \
+  -F "workflow_engine=cwltool" \
   -F "workflow_engine_parameters=${workflow_engine_parameters}" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)
 

tests/curl_example/nextflow_file_input.sh

@@ -23,7 +23,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=NFL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./file_input.nf" \
-  -F "workflow_engine_name=nextflow" \
+  -F "workflow_engine=nextflow" \
   -F "workflow_attachment=@${workflow}" \
   -F "workflow_attachment=@${input_file}" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)

tests/curl_example/nextflow_file_input_with_docker.sh

@@ -27,7 +27,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=NFL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./file_input.nf" \
-  -F "workflow_engine_name=nextflow" \
+  -F "workflow_engine=nextflow" \
   -F "workflow_engine_parameters=${workflow_engine_parameters}" \
   -F "workflow_attachment=@${workflow}" \
   -F "workflow_attachment=@${input_file}" \

tests/curl_example/nextflow_params_outdir.sh

@@ -23,7 +23,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=NFL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./params_outdir.nf" \
-  -F "workflow_engine_name=nextflow" \
+  -F "workflow_engine=nextflow" \
   -F "workflow_attachment=@${workflow}" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)
 

tests/curl_example/nextflow_params_outdir_with_docker.sh

@@ -27,7 +27,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=NFL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./params_outdir.nf" \
-  -F "workflow_engine_name=nextflow" \
+  -F "workflow_engine=nextflow" \
   -F "workflow_engine_parameters=${workflow_engine_parameters}" \
   -F "workflow_attachment=@${workflow}" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)

tests/curl_example/nextflow_str_input.sh

@@ -22,7 +22,7 @@ response=$(curl -fsSL -X POST \
   -F "workflow_type=NFL" \
   -F "workflow_type_version=1.0" \
   -F "workflow_url=./str_input.nf" \
-  -F "workflow_engine_name=nextflow" \
+  -F "workflow_engine=nextflow" \
   -F "workflow_attachment=@${workflow}" \
   http://${SAPPORO_HOST}:${SAPPORO_PORT}/runs)