Description
Describe the bug
Assuming a workflow is launched with signalWithStart, the following workflow code in TS SDK 1.11.3 or earlier would result in sequence A-B-C. Same in later releases if not using the OTel workflow interceptor. However, in 1.11.5+ with OTel, we’d get A-C-B (i.e. equivalent of signal coming in a later WFT).
export async function myWorkflow(name: string): Promise<void> {
log('A');
setHandler(defineSignal('mySignal'), async () => {
log('B');
});
log('C');
}
This change is due to addition of tracing propagation on inbound signals in 1.11.5 (#1449). Order of promise completion on outbound scheduleLocalActivity has similarly been modified in 1.11.6 (#1577). Both of these changes may result in Non-Determinism errors when older workflows are replayed on newer release of the SDK with the OTel interceptor installed.
For context, the changes mentioned previously added a certain number of await statements to the corresponding code paths, so the actual code is now executed as a microtask rather than synchronously, hence the change in execution order. As a general rule, interceptors provided by the SDK itself should as much as possible avoid adding yield points. We should also document this risk on the Workflow Interceptor types, as user-implemented interceptors may face similar issues.