Commit 06b4a29
authored
docs(config): document grpc.attempts timing + tuning guidance (#10868)
The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.
Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
140, attempts_sleep_time 5), with a note that inference-timeouts and
the watchdog are unaffected.1 parent e62221b commit 06b4a29
1 file changed
Lines changed: 33 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
914 | 914 | | |
915 | 915 | | |
916 | 916 | | |
917 | | - | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
918 | 922 | | |
919 | | - | |
920 | | - | |
921 | | - | |
922 | | - | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
923 | 951 | | |
924 | 952 | | |
925 | 953 | | |
| |||
0 commit comments