Skip to content

agents: implement gRPC retry policies #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
santigimeno wants to merge 2 commits into
base: node-v24.x-nsolid-v6.x
Choose a base branch
from
+567 −41
Open

agents: implement gRPC retry policies #387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b76d37e
agents: implement gRPC retry policies
santigimeno Jan 19, 2026
3a796c9
test: add gRPC retry policy tests
santigimeno Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions agents/grpc/src/grpc_agent.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ const int CONSOLE_ID_SIZE = 36;

const seconds DEFAULT_GRPC_TIMEOUT = seconds{ 60 };

// Retry policy configuration
constexpr size_t retry_max_attempts = 5;
constexpr auto retry_initial_backoff = std::chrono::milliseconds(500);
constexpr auto retry_max_backoff = std::chrono::seconds(5);
constexpr float retry_backoff_multiplier = 2.0f;
Comment on lines +83 to +87
Copy link

@coderabbitai coderabbitai bot Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

cat -n agents/grpc/src/grpc_agent.cc | sed -n '75,95p'

Repository: nodesource/nsolid

Length of output: 874

🏁 Script executed:

cat -n agents/grpc/src/grpc_agent.cc | sed -n '1070,1085p'

Repository: nodesource/nsolid

Length of output: 797

🏁 Script executed:

# Search for any retry-related documentation or configuration in the codebase
rg -i "maxattempts|retry.*semantic|retry.*policy" --type cc --type h -A 3 -B 3

Repository: nodesource/nsolid

Length of output: 87

🌐 Web query:

gRPC maxAttempts semantics does it include initial attempt

💡 Result:

In gRPC retry policy semantics, maxAttempts includes the initial (original) RPC attempt—it is the total number of tries, not “retries only.” (gitea.cncfstack.com)

So, for example, maxAttempts: 3 means 1 initial attempt + up to 2 retries. (gitea.cncfstack.com)

Citations:

🌐 Web query:

OpenTelemetry C++ OtlpGrpcClientOptions retry maxAttempts documentation

💡 Result:

In opentelemetry-cpp (OTLP gRPC exporter), the equivalent of gRPC retry policy MaxAttempts is exposed on OtlpGrpcClientOptions as:

  • retry_policy_max_attempts (default 5)
  • plus backoff knobs: retry_policy_initial_backoff (default 1s), retry_policy_max_backoff (default 5s), retry_policy_backoff_multiplier (default 1.5) (deepwiki.com)

You can also set it via environment variable:

  • OTEL_CPP_EXPORTER_OTLP_TRACES_RETRY_MAX_ATTEMPTS (default "5") (deepwiki.com)

Notes:

  • Retry handling for OTLP/gRPC in C++ was introduced as an experimental feature gated by WITH_OTLP_RETRY_PREVIEW. (github.com)
  • The OTLP spec recommends retrying only for certain retryable gRPC status codes and using exponential backoff. (opentelemetry.io)

Citations:

Clarify intent: does retry_max_attempts = 5 mean 5 total attempts or 5 retries?

In gRPC, maxAttempts counts the total number of attempts (including the initial one). So retry_max_attempts = 5 means 1 initial attempt + up to 4 retries, not 5 retries. If the intent is 5 retries, the value should be 6. Consider adding a clarifying comment to prevent confusion.

🤖 Prompt for AI Agents 
In `@agents/grpc/src/grpc_agent.cc` around lines 83 - 87, The constant
retry_max_attempts currently set to 5 is ambiguous because gRPC's maxAttempts
counts total attempts (initial + retries); decide whether you intended 5 total
attempts or 5 retries—if you meant 5 retries, change retry_max_attempts to 6;
otherwise add a clarifying comment next to retry_max_attempts explaining "gRPC
counts this as total attempts (initial + retries), so 5 means 1 initial + 4
retries)". Update the constant or add the comment near retry_max_attempts (and
keep existing names retry_initial_backoff, retry_max_backoff,
retry_backoff_multiplier unchanged).


JSThreadMetrics::JSThreadMetrics(SharedEnvInst envinst):
metrics_(ThreadMetrics::Create(envinst)) {
}
Expand Down Expand Up @@ -1068,6 +1074,11 @@ int GrpcAgent::config(const json& config) {
}
}

opts.retry_policy_max_attempts = retry_max_attempts;
opts.retry_policy_initial_backoff = retry_initial_backoff;
opts.retry_policy_max_backoff = retry_max_backoff;
opts.retry_policy_backoff_multiplier = retry_backoff_multiplier;

nsolid_service_stub_ =
GrpcClient::MakeNSolidServiceStub(opts, tls_keylog_file_);

Expand Down
34 changes: 34 additions & 0 deletions agents/grpc/src/grpc_client.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,40 @@ std::shared_ptr<Channel>
grpc_arguments.SetInt(GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS, 1);
grpc_arguments.SetInt(GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA, 0);

static const auto kServiceConfigJson = std::string_view{R"(
{
"methodConfig": [
{
"name": [{}],
"retryPolicy": {
"maxAttempts": %0000000000u,
"initialBackoff": "%0000000000.1fs",
"maxBackoff": "%0000000000.1fs",
"backoffMultiplier": %0000000000.1f,
"retryableStatusCodes": [
"UNAVAILABLE"
]
}
}
]
})"};

// Allocate string with buffer large enough to hold the formatted json config
auto service_config = std::string(kServiceConfigJson.size(), '\0');
float initial_backoff = options.retry_policy_initial_backoff.count();
float max_backoff = options.retry_policy_max_backoff.count();
float backoff_multiplier = options.retry_policy_backoff_multiplier;
std::snprintf(
service_config.data(),
service_config.size(),
kServiceConfigJson.data(),
options.retry_policy_max_attempts,
std::min(std::max(initial_backoff, 0.f), 999999999.f),
std::min(std::max(max_backoff, 0.f), 999999999.f),
std::min(std::max(backoff_multiplier, 0.f), 999999999.f));

grpc_arguments.SetServiceConfigJSON(service_config);

return CreateCustomChannel(options.endpoint,
MakeCredentials(options, tls_keylog_file),
grpc_arguments);
Expand Down
4 changes: 3 additions & 1 deletion deps/opentelemetry-cpp/otlp-http-exporter.gyp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,19 +64,21 @@
'ENABLE_ASYNC_EXPORT',
'ENABLE_OTLP_GRPC_CREDENTIAL_PREVIEW',
'OPENTELEMETRY_STL_VERSION=2020',
'ENABLE_OTLP_RETRY_PREVIEW',
],
'dependencies': [
'../protobuf/protobuf.gyp:protobuf',
'../curl/curl.gyp:curl',
'../grpc/grpc.gyp:grpc++',
'../protobuf/abseil.gyp:abseil_proto',
'../protobuf/abseil.gyp:abseil_proto',
'../zlib/zlib.gyp:zlib',
],
'direct_dependent_settings': {
'defines': [
'ENABLE_ASYNC_EXPORT',
'ENABLE_OTLP_GRPC_CREDENTIAL_PREVIEW',
'OPENTELEMETRY_STL_VERSION=2020',
'ENABLE_OTLP_RETRY_PREVIEW',
],
'include_dirs': [
'api/include',
Expand Down
Loading
Loading