Skip to content

Commit 06b4a29

Browse files
authored
docs(config): document grpc.attempts timing + tuning guidance (#10868)
The gRPC configuration table only listed the two fields with a one-line description each, without defaults, without explaining what the total load window looks like, and without hinting when a user should adjust them. In practice the default 20 attempts x 2 s = 40 s window is way too tight for large NVFP4 / FP8 models on slow storage or first-run CUDA-graph capture, and the resulting kill (exitCode=120, 'context canceled') looks like a backend crash even though the backend is still making legitimate forward progress. Extend the section with: - Defaults column (20 and 2) added to the table - Prose explaining that these govern the readiness handshake between LocalAI and a freshly spawned backend (Health polling loop) - Total-load-window formula - Concrete failure signature so users can recognize a timeout-kill vs. a real backend crash - Example configuration for a ~10 min cold-load window (grpc.attempts 140, attempts_sleep_time 5), with a note that inference-timeouts and the watchdog are unaffected.
1 parent e62221b commit 06b4a29

1 file changed

Lines changed: 33 additions & 5 deletions

File tree

docs/content/advanced/model-configuration.md

Lines changed: 33 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -914,12 +914,40 @@ Define pipelines for audio-to-audio processing and the [Realtime API]({{%relref
914914

915915
## gRPC Configuration
916916

917-
Backend gRPC communication settings:
917+
Backend gRPC communication settings. These control the readiness handshake
918+
between LocalAI and a freshly spawned backend process — LocalAI polls the
919+
backend's `Health` gRPC method up to `grpc.attempts` times, sleeping
920+
`grpc.attempts_sleep_time` seconds between polls, before giving up and
921+
terminating the backend as unresponsive.
918922

919-
| Field | Type | Description |
920-
|-------|------|-------------|
921-
| `grpc.attempts` | int | Number of retry attempts |
922-
| `grpc.attempts_sleep_time` | int | Sleep time between retries (seconds) |
923+
| Field | Type | Default | Description |
924+
|-------|------|---------|-------------|
925+
| `grpc.attempts` | int | 20 | Number of health-check attempts before the backend is killed as unresponsive |
926+
| `grpc.attempts_sleep_time` | int | 2 | Sleep time between health-check attempts (seconds) |
927+
928+
**Total load window ≈ `grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)`.**
929+
The default of `20 × 2 s ≈ 40 s` is fine for typical backends but is too
930+
short for large models that need substantial time to become gRPC-ready
931+
after the process starts — for example NVFP4 / FP8 models whose shard
932+
loading and CUDA-graph capture can take several minutes, or slow storage
933+
backends. If the backend keeps getting killed while still legitimately
934+
loading (visible as `exitCode=120` + `rpc error: code = Canceled desc =
935+
context canceled` in the LocalAI log, while the backend's own stderr
936+
shows continued forward progress), raise these values.
937+
938+
Example configuration for a model that needs up to ~10 minutes to become
939+
gRPC-ready (large NVFP4 model, cold shard load + CUDA-graph capture):
940+
941+
```yaml
942+
grpc:
943+
attempts: 140
944+
attempts_sleep_time: 5
945+
```
946+
947+
This gives a ~700 s window while keeping health-check polling frequent
948+
enough to detect real backend crashes quickly. The values only affect
949+
the initial readiness handshake — inference-request timeouts and the
950+
watchdog are unchanged.
923951

924952
## Overrides
925953

0 commit comments

Comments
 (0)