Probable SSM ThrottlingException: Rate exceeded with standard queues from burst-y traffic

[tkovis]

Describe the issue
When publishing a large number of payloads concurrently without waiting for the first to finish, you can get throttled unexpectedly even if you use a plugin to change the default fifo queues into standard queues where throttling should not happen.

  await Promise.all(
    largeArray.map(async (payload) =>
      arc.queues.publish({
        name: "queue-name",
        payload,
      })
    )
  );

{
    "errorType": "ThrottlingException",
    "errorMessage": "Rate exceeded",
    "name": "ThrottlingException",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 400,
        "requestId": "...",
        "attempts": 3,
        "totalRetryDelay": 101
    },
    "__type": "ThrottlingException",
    "message": "Rate exceeded",
    "stack": [
        "ThrottlingException: Rate exceeded",
        "    at throwDefaultError (/var/task/index.js:6319:24)",
        "    at /var/task/index.js:6328:9",
        "    at de_CommandError (/var/task/index.js:25195:18)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/task/index.js:756:24",
        "    at async /var/task/index.js:872:22",
        "    at async /var/task/index.js:10952:42",
        "    at async /var/task/index.js:494:26"
    ]
}

It seems that this is caused by the service discovery only caching after it has succesfully fetched which causes every call to publish launch a concurrent fetch for service discovery and eventually causes a throttling error.

Steps to reproduce
Steps to reproduce the behavior:

1. Use a plugin to change the queue to a standard queue
2. Try to publish a large number of payloads concurrently

Expected behavior
If this is caused by service discovery not caching the initial promise, then it is expected that the first call to publish causes a fetch for service discovery and every other call to publish waits until it resolves and does not try to fetch things by themselves.
const store = { cache: null };

const fetchOrUseCached = () => {
  if (store.cache) return store.cache;
  // immediately store the promise for next synchronous call so the same promise is used for every request
  store.cache = fetchSomething();
  return store.cache;
}

I am unsure about what actually is the cause, this is just a guess. I am also unsure if this would be the correct solution.

[filmaj]

Hmm, so are you saying the throttle exception comes from the service discovery and not from the queue publish? Too bad that's not obvious from the stack trace.. we should improve the error reporting there, too.

What version of functions are you using?

[tkovis]

Hmm, so are you saying the throttle exception comes from the service discovery and not from the queue publish

This was our hypothesis. As a workaround we decided to do a single publish first, wait for it finish and then do the rest. The issues stopped after so I think it likely is not a throttling error related to SQS standard queue publish but something that arc.queues.publish does before the actual sqs publish.

Also we have made a custom plugin to modify the cloudformation, use it to change to queue to a standard queue and have verified through console that it was indeed a standard queue (which should practically never throttle afaik).

Too bad that's not obvious from the stack trace

Idd we used a lot of time trying to find out how a standard queue publish can cause a ThrottlingException .

What version of functions are you using?

"@architect/functions": "^7.0.0"

[filmaj]

Is there any additional info from CloudWatch that you can discern? For example, does the occurrence of the throttle exception in logs coincide with throttle events in Lambda, or any other kind of events or metrics from Lambda, or SQS, or SSM? That may give us a hint as to where this is coming from.

Quick sanity check: do you have concurrency or other limits set on the queue or the lambda it triggers? That could also cause throttling.

I would not be surprised if this exception comes from SSM, especially during burst-y traffic, which would result in many cold starts / new VMs being spun up, meaning the SSM cache would not be in place for the fresh VM, meaning SSM would be hit anew by each new concurrent lambda spinning up. I mentioned as much as a possibility (and saw it happen once or twice with a client project) way back in July 2023 in architect/architect#1435. How many resources are in your stack? The larger the stack, the more services there are to discover, the more SSM pages need to be retrieved, the higher the likelihood of hitting SSM RPS limits. Back in 2023, SSM had a 40 RPS limit. IIRC a page of SSM parameters returns a max of 10 items (!) with a 100ms latency. Combine those numbers with a rough estimate of traffic and how many SSM parameters are needed to provide service discovery in your app, and you could model different types of burst traffic that would trigger SSM throttling exceptions.

[filmaj]

One more idea: if you could upgrade to arc functions v8+, aws-lite would power the underlying AWS interactions, which probably would provide better stack traces, which may give additional clues.

[tkovis]

Is there any additional info from CloudWatch that you can discern

In the last 3 months there have been 0 lambda throttles and a maximum of 1 concurrent execution.

do you have concurrency or other limits set on the queue or the lambda it triggers

No concurrency limits on the lambda. The queue is a standard queue with no limits set in place. The largest queue "
Number Of Messages Received" in the last 3 months is 3.3k. It is possible that a single lambda invocation has published the total 3.3k messages.

How many resources are in your stack?

20 parameters in SSM, 1 table, 2 http handlers, 1 queue handler, 2 scheduled handlers

One more idea: if you could upgrade to arc functions v8+, aws-lite would power the underlying AWS interactions, which probably would provide better stack traces, which may give additional clues

Yes we will eventually do this, but I don't think it will help since the exceptions have stopped occurring since we started doing a single publish before doing the rest and we are unable to reproduce it in testing environments. If we can reproduce the exception, we will upgrade if we haven't already

  const [first, ...rest] = parsed.data;
  await arc.queues.publish({
    name: "queue-name",
    payload: first,
  });
  await Promise.all(
    rest.map(async (payload) =>
      arc.queues.publish({
        name: "queue-name",
        payload,
      })
    )
  );

[filmaj]

Fair enough. Thanks for sharing your workaround, that is helpful to know.. though that doesn't make sense to me how that would avoid hitting the SSM rate limit by burst traffic creating many cold lambdas that each have to query SSM to complete service discovery.

I will rename the issue title, link it back to architect/architect#1435 in its post, and close this issue. Feel free to re-open the issue or post a new one if you think there's more to discuss or address here, but AFAICT, if this is indeed due to SSM rate limits, there is not much we can do here unless we address changing the architecture of the service discovery system as I suggested in architect/architect#1435.