Enforce max timeout for AgentCore invoke call

[justinschoeff]

The AgentCore API is synchronous and while it does support streaming
responses, that is left up to the runtime code itself. This means the
invocation request can hang up to 8 hours!

Even when runtimes last a minute it is problematic for WCI. This commit
aims to limit the allowed runtime to 5 seconds and assuming success.

What was changed

Add a timeout via context to the AWS AgentCore SDK InvokeAgentRuntime call to prevent it from hitting the activity timeout. This also adds an http trace on request write as a gauge to whether the invocation was actually transmitted or not.

Why?

Without this, the runtime can hit the activity timeout causing retries and create worker-deployment failures. Additionally, each call (including retries) does spin up a new worker agentcore side.

Checklist

1. Closes

2. How was this tested:
   Launched an agent runtime that exceeded the 60s activity timeout and saw success alongside a consistent 5s between activity start to completed (hitting the agentCoreMaxRuntime.

Launched an agent runtime that responded immediately and saw success.

3. Any docs updates needed?

[github-actions]

Merging this branch will not change overall coverage

Impacted Packages	Coverage Δ	🤖
github.com/temporalio/temporal-auto-scaled-workers/wci/workflow/compute_provider	0.00% (ø)

---

Coverage by file

Changed files (no unit tests)

Changed File	Coverage Δ	Total	Covered	Missed	🤖
github.com/temporalio/temporal-auto-scaled-workers/wci/workflow/compute_provider/aws_agentcore.go	0.00% (ø)	0	0	0

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

• github.com/temporalio/temporal-auto-scaled-workers/wci/workflow/compute_provider/aws_agentcore_test.go

[mahars80]

Does this feel any aggressive? Also, a noob question, why does this not meet the bar to be a config that can be changed at runtime? :)

[justinschoeff]

It is a bit aggressive. The only way this really becomes a problem is if the invoke call from Temporal server to AWS takes 5s before sending the request, which shouldn't be the case and usually would be transient. The activity here would retry and hopefully succeed. I'm not completely opposed to making configurable server-side (not WDV/user configurable), but I'm not sure it's worth it.