v1.27.0

Schema Changes

Before upgrading your Temporal Cluster to v1.27.0, you must upgrade your core schema if you are using MySQL or PostgreSQL, and your visibility schema to the following:

• Core:
  • MySQL schema v1.15
  • PostgreSQL schema v1.15
• Visibility:
  • Elasticsearch schema v9
  • MySQL schema 1.9
  • PostgreSQL schema v1.9

Please see our upgrade documentation for the necessary steps to upgrade your schemas.

NOTE: The upgrade to MySQL and PostgreSQL Visibility schemas may come with temporary performance degradation because of creation of a new column _version which has default values. Consider performing the schema upgrades when load is low. There are protective mechanisms in place to account for timeouts from any VisibilityStore.

Deprecation of Visibility Scan APIs

Visibility Scan APIs have been deprecated in favor of List Workflow APIs. Visibility Scan APIs will be removed in a future version. Migration to List Workflow APIs will be required in future versions.

Nexus GA

Nexus is now GA with a stable server API.

Read more here on how to disable Nexus or how to operate it here.

Notable features and bug fixes since v1.26.2:

• The server now allows a maximum of 30 pending Nexus operations per workflow by default, as opposed to the previous limit being 30 total.
• Add support for attaching callbacks, request IDs, and links to a workflow via StartWorkflowExecutionRequest.OnConflictOptions with WorkflowIdConflictPolicy of USE_EXISTING. This allows multiple operations to be backed by the same workflow.
• Misc small features and tests.

Safe Deploys

The following APIs are added for Worker Versioning. All APIs are experimental and not yet recommended for production usage. You need to set the dynamic configs system.enableDeployments and system.enableDeploymentVersions in order to use them.

• ListWorkerDeployments
• DescribeWorkerDeployment
• DescribeWorkerDeploymentVersion
• SetWorkerDeploymentCurrentVersion
• SetWorkerDeploymentRampingVersion
• UpdateWorkerVersionMetadata
• DeleteWorkerDeployment
• DeleteWorkerDeploymentVersion

The following APIs are now deprecated and replaced by above:

• DescribeDeployment
• ListDeployments
• GetDeploymentReachability
• GetCurrentDeployment
• SetCurrentDeployment

Activity Commands (pre-release)

Changes to the Activity Commands — a set of APIs designed to resolve issues related to activity execution. The following APIs where updated:

• UpdateActivityOptionsById was renamed to UpdateActivityOptions. This API can be used by the client to update the options of an activity while activity is running.
• PauseActivityById was renamed to PauseActivity. With this API, If the Activity is not currently running (e.g. because it previously failed and is waiting for the next retry), it will not be run again until it is unpaused.
  • activity_type parameter was added. If this parameter is set - all running activities of this type will be paused.
• UnpauseActivityById was renamed to UnpauseActivity. With this API clients can re-schedule a previously-paused Activity for execution.
  • no_wait parameter was removed
  • activity_type parameter was added. If this parameter is set - all paused activities of this type will be unpaused.
  • match-all parameter was added. If this parameter is set - all paused activities will be unpaused.
  • jitter parameter was added. If set, the activity will start at a random time within the specified jitter duration.
• ResetActivityById was renamed to ResetActivity. With this API clients can unpauses the execution of a previously paused activity, specified by its ID.
  • no_wait parameter was removed
  • activity_type parameter was added. If this parameter is provided - all paused activities of this type will be unpaused.
  • keep_paused parameter was added. If this parameter is provided - all paused activities will stay paused. Otherwise they will be unpaused.
  • jitter parameter was added. If set, the activity will start at a random time within the specified jitter duration.
• New batch operation was introduced - BatchOperationUnpauseActivities.

Workflow Reset with children

In this release we have added the functionality to reset a workflow with a pending child.

Prior to this release reseting to a point between child workflow initiated and child workflow completed was not supported (the reset operation would fail). In the current release the reset operation will allow this case and the behavior of the parent after reset is to reconnect to the running child. The new run of the parent will receive the child’s completion event and result (if any) from the child.

The feature is gated behind a per namespace boolean dynamic configuration AllowResetWithPendingChildren which is enabled by default for all namespaces.

Note: If you are using Go-SDK and are relying on the SDK to generate child workflow IDs then you need to update it to the latest version to be able to use this feature. Other SDKs don’t need any upgrade to use this feature.

Delete namespace improvement

1. Delete workflow executions RPS is now dynamic. Previously, frontend.deleteNamespaceDeleteActivityRPS was read only once when namespace deletion started, and subsequent changes to this dynamic config didn't affect the ongoing deletion. This was inconvenient for large namespaces since the default RPS is only 100. Now the RPS can be adjusted on the fly.

   Please note: Since deletion of Workflow Execution is an asynchronous process, this RPS controls the rate at which delete execution tasks are created. Decreasing this value (for example, from 1000 to 10) won't immediately slow down the process, as existing tasks in the transfer queue must be processed first.

2. DeleteExecutionsWorkflow now supports a stats query to track its progress. Since this Workflow can run for hours after a namespace is marked as deleted, it was previously difficult to monitor how many Workflow Executions remained. The new query handler provides current statistics about total and remaining executions.

3. Metrics and logging have been enhanced for better actionability. Key improvements include:

   1. Monitor ReclaimResources workflow failures using the metrics reclaim_resources_namespace_delete_failure and reclaim_resources_delete_executions_failure.
   2. Track DeleteExecutionsWorkflow progress using the metrics delete_executions_success and delete_executions_failure.

4. Business critical namespaces can be protected from deletion. Use dynamic config to list namespaces which are not deletable:

   worker.protectedNamespaces:
     - value:
         - critical_namespace
         - just_very_important_namespace

5. Sleep duration in ReclaimResourcesWorkflow now supports dynamic changes. If a namespace delete delay was mistakenly set too long, you can now modify it after the Workflow has started. Use this command to update the delay to a new value (10 hours in this example):

   temporal workflow update execute --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"10h"'
   

   Or use this command to remove the delay entirely:

   temporal workflow update execute --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"0"'
   

   Please note: The new delay starts from when it is set, not from when the original timer was created. For example, if the Workflow has already slept for 2 hours and the timer is updated to 10h, it will sleep for another 10 hours, not 8.

Scheduled Actions

• Scheduler workflow version has been updated.
  • FutureActionTimes now accounts for a schedule’s update time and RemainingActions.
  • ScheduleActionResult now includes a WorkflowExecutionStatus field, providing an eventually-consistent view of a workflow’s status within List results.
• Bugfix: Queries on schedules might have been delayed after dynamic config changes were applied to per-namespace workers. An anti-entropy mechanism has been added to the per-namespace worker component.

Fix out-of-order Visibility tasks with SQL database

All SQL stores used for Visibility had the rare possibility to perform updates to a workflow's visibility state out-of-order. This could result in Workflows occasionally appearing to have state that is out of date.

Implementation

This has been fixed by adding a new column _version to all SQL store implementations. Queries to update Visibility data now ensure the _version advances before performing any writes.

Updates to Visibility data are prepared by checking actual Workflow state. Therefore when a write is rejected for being out-of-order, we know the VisibilityStore already contains a equal or more up-to-date state, so we drop out-of-order updates silently.

Important Notes

• Make sure to upgrade your Schemas before advancing your temporal version. See the Schema Changes section.
• This will likely reduce the performance of SQL VisibilityStores.
• We suggest ElasticSearch as the performant solution for VisibilityStore.

---

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.27.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Full Changelog: v1.26.2...v1.27.0