Skip to content

KLAUD_DEBUG.md

Latest commit

 

History

History
283 lines (208 loc) · 16 KB

KLAUD_DEBUG.md

File metadata and controls

283 lines (208 loc) · 16 KB
Raw

KLAUD_DEBUG.md — Operational Knowledge for Recipe-Bump PRs

A running playbook of failures the Klaud-Cold image-bump cron has hit, the diagnoses, and the fixes/workarounds applied. Read this first before debugging a new failing claude/* PR — most failure modes here recur.

When you fix something not yet listed, add it here so the next session doesn't re-learn it.

1. PR setup-stage failures

1.1 perf-changelog.yaml: deletion-not-allowed

Symptom: the setup job fails before any sweep runs with

ValueError: Deletions are not allowed in /home/runner/work/InferenceX/InferenceX/perf-changelog.yaml.
Only additions to the changelog are permitted. Found deleted line: ...

Root cause: Cron-PR branches go stale; when main merges new changelog entries, the PR's local snapshot of perf-changelog.yaml no longer covers them, so the validator sees the missing lines as deletions. A naive rebase can also strip trailing whitespace from unrelated entries — same effect (e.g. pr-link: ...1311 → pr-link: ...1311).

Fix (canonical):

# In the PR's worktree, after `git merge origin/main` conflicts on perf-changelog.yaml:
git checkout origin/main -- perf-changelog.yaml          # take main's bytes verbatim
cat >> perf-changelog.yaml <<EOF                          # then append THIS PR's entry at tail

- config-keys:
    - <recipe-key>
  description:
    - "<one-line summary>"
  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/<N>
EOF
python3 -c "import yaml; yaml.safe_load(open('perf-changelog.yaml'))"

Do not try a 3-way merge of perf-changelog.yaml — whitespace edits will silently re-trigger the deletion check.

After committing and pushing the resolution, the synchronize run checks the changelog with the same matrix processor used by setup, then checks the reuse authorization. This catches deleted history or malformed appended entries before reuse can skip setup. utils/merge_with_reuse.sh <PR> performs the push and waits for the PR checks automatically.

2. vLLM v0.21.x / v0.20.x: GPU OOM at model-load

Symptom: vLLM workers OOM during weight loading or right after warmup:

  • HSA_STATUS_ERROR_OUT_OF_RESOURCES: Available Free mem : 0 MB (AMD)
  • torch.OutOfMemoryError: CUDA out of memory. ... GPU N has X GiB of which Y MiB is free (NVIDIA)
  • vLLM may also log _check_enough_kv_cache_memory failing with negative available bytes (e.g. -25.24 GiB).

Root cause: v0.21.0 (and v0.20.2+) enabled an aggressive CUDA-graph memory profiler that pre-reserves a large chunk of VRAM up front (~30% on B200), shrinking effective --gpu-memory-utilization well below what the flag says. Old SHA-pinned custom images had a smaller footprint, so the recipe's existing 0.95 setting now starves the KV cache.

Fix: in benchmarks/single_node/<recipe>.sh, either:

  1. Lower --gpu-memory-utilization (0.95 → 0.90, sometimes 0.85). Matches the H100/H200/B200 NVIDIA pattern. Smallest blast radius.
  2. Disable the profiler entirely for cases where lowering isn't enough: export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 before vllm serve. Matches benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:65.

Seen on: #1395 (kimik2.5-fp4-b200-vllm — needed env var), #1403 (gptoss-fp4-mi300x-vllm — needed 0.90), #1461 (dsv4-fp8-h200-vllm — needed 0.90).

2.1 DEP CUDA-graph capture OOM on GB300

Symptom: TP1 + data/expert-parallel decode workers load successfully and allocate the KV cache, then fail in breakable_cudagraph.py at torch.cuda.graph.capture_end() with CUDA error: out of memory. Large GB300 VRAM does not prevent this because vLLM fills the configured memory budget with KV cache before capturing hundreds of persistent graphs.

Root cause: max-num-seqs and max-cudagraph-capture-size were sized from global benchmark concurrency instead of per-DP-rank concurrency. MiniMax-M3 DEP4/DEP8 recipes requested capture sizes of 4096-8192 and up to 4096 sequences, creating 358-806 graphs per GPU.

First-line tuning: keep gpu-memory-utilization: 0.90, but size graph limits to the per-rank load. For the GB300 MiniMax-M3 sweep, use max-num-seqs: 512 and max-cudagraph-capture-size: 2048 on DEP decoders. This matches the single-node GB300 recipe and still covers the largest 512 requests per DP rank. If capture still OOMs, lower decode gpu-memory-utilization to 0.85.

Seen on: #1735 (MiniMax-M3 MXFP8 GB300 dynamo-vLLM).

3. Custom DSV4 image → generic v0.5.12 OOMs

Symptom: DSV4 recipes work on their SHA-pinned lmsysorg/sglang:deepseek-v4-hopper@sha256:... (or deepseek-v4-b300, deepseek-v4-blackwell) custom builds, but OOM on weights load when bumped to the generic v0.5.12-cu130 release tag. Example: DSV4-Pro FP8+MTP weights consume ~125.43 GB / 141 GB per H200, leaving -4.05 GB for KV cache.

Root cause: The custom DSV4 images use a different weight layout / EAGLE draft handling that fits in less memory than the generic release. The release tag isn't a drop-in replacement.

Fix: keep DSV4 recipes pinned to their custom SHA-pinned image until upstream sglang gains the same DSV4-specific weight handling. Bumping to the generic tag is currently NOT viable.

Seen on: #1460 (dsv4-fp8-h200-sglang+mtp).

4. Upstream sglang v0.5.12 B300 regressions

Three distinct upstream regressions on NVIDIA B300 (Blackwell Ultra, sm_103 — compute capability 10.3) shipped in lmsysorg/sglang:v0.5.12-cu130. (sm_120 is for consumer Blackwell / RTX 50 series, not B300 — don't propagate that.)

4a. DeepGemm TMA-descriptor crash (GLM-5-FP8)

Symptom: CUDA graph capture aborts with CUDA_ERROR_ILLEGAL_ADDRESS (700) at /deepgemm/csrc/.../runtime_utils.hpp:143 on the first batch size for every TP rank. Server never serves a prompt.

Workarounds (any one):

  1. --fp8-gemm-runner-backend cutlass to bypass DeepGemm via CUTLASS.
  2. export SGL_ENABLE_JIT_DEEPGEMM=0 before python -m sglang.launch_server to skip JIT DeepGemm.
  3. Pin recipe to lmsysorg/sglang:v0.5.11-cu130.

Filed upstream: sgl-project/sglang#25551. Seen on #1421.

4b. trtllm GEMM bug at bs=128 + MTP / EAGLE (GLM-5-NVFP4)

Symptom: EAGLE draft CUDA graph capture crashes immediately at the largest batch size with RuntimeError ... trtllm_batched_gemm_runner.cu:276 ... numBatches=256, GemmMNK 128x1024x6144. The target model captures fine; only the draft model crashes.

Workarounds:

  1. Cap --cuda-graph-max-bs and --max-running-requests to 64 in the launch script to avoid the bs=128 trigger.
  2. Comment out the MTP/EAGLE scenarios on B300 in the recipe.
  3. Pin to v0.5.11-cu130.

Filed upstream: sgl-project/sglang#25563. Seen on #1420.

4c. flash_attn SM-arch assertion (qwen3.5-bf16)

Symptom: All 4 TP workers AssertionError on first forward pass:

File "/opt/venv/.../sglang/srt/layers/attention/flashattention_backend.py:..."
  assert sm_100 <= arch <= sm_110f

B300 is sm_103 (compute capability 10.3, Blackwell Ultra) — which is nominally inside the asserted sm_100..sm_110f range, yet the assertion still fires. Best guess is the cute kernel's Arch.sm_110f set only matches the architecture-specific feature-flag variants it was compiled for (e.g. sm_100, sm_100f, sm_110, sm_110f) and sm_103 / sm_103a isn't in that explicit list. Server never becomes healthy; warmup times out at 600s.

Fix: Needs an sglang image with flash_attn that recognises sm_103 / sm_103a — no local workaround. Pin to v0.5.11-cu130 in the meantime.

Seen on #1422.

5. Cluster infrastructure (AMD MI355X / MI300X / MI325X)

5.1 mia1-p01-g09 / g19 / g37 (amd-tw-mi355) — persistently drained

  • g09: pyxis is broken
  • g19: Kill task failed (JobId=N StepId=N)
  • g37: permission issues with GHA runner workflows : Not responding (down since Mar 2026)

If a sweep job lands on any of these, it'll never start. Nothing to do at the recipe level — these stay drained until ops fixes them.

5.2 mia1-p01-g11 / g12 / g31 — docker socket perms

Symptom: mi355x jobs fail with permission denied while trying to connect to the docker API at unix:///var/run/docker.sock during the docker stop $(docker ps -a -q) cleanup step, cascading into SLURM job expiration. Fix: ops needs to fix docker group / socket perms on these nodes. Recipe-level workaround: none.

5.3 chi-mi300x-049 — /nvme_home disk-full

Symptom: pyxis container extraction fails with No space left on device writing to /nvme_home/gharunner/.local/share/enroot/pyxis_*/opt/rocm-*/.... The /nvme_home partition is hosted under / on this node and has been chronically near-full.

Fix already landed: runners/launch_mi300x-amds.sh now pins salloc to only known-good mi300x nodes (chi-mi300x-[034-036,054,057-058]) — see PR #1462. chi-mi300x-049 is held in State=DOWN by a watchdog on the controller (/home/gharunner/_audit/drain_049_watchdog.sh) that re-applies the drain every 10s if SLURM auto-clears it (which it does on dynamic-norm nodes).

5.4 chi-mi325x-pod1-017 — orphaned port-8888 process

Symptom: sglang server bind fails with [Errno 98] Address already in use on port 8888. Held by an MLPerf accuracy run started outside SLURM. Fix: SSH to controller, find the holder via ss -tlnp | grep :8888, kill the PID. If recurring, file with the team running MLPerf experiments.

5.5 Cluster controller layout

  • amd-vultr-mi300: SLURM controller for 7 mi300x nodes (3 down, see 5.3).
  • amd-vultr-mi325: SLURM controller for 6 mi325x nodes.
  • amd-tw-mi355: jumpbox → ssh further to compute nodes (mia1-p01-gNN). 12 nodes (3 drained, see 5.1).
  • /home is NFS-mounted across clusters from chi-mi325x-pod1-001:/nfs/homes, root-writable.
  • /tmp and /nvme_home are per-node local; HF cache lives at node-local /raid/hf-hub-cache/ (2.7T per mi300x node).
  • Use srun -w <FQDN> (with the full FQDN, not the short hostname) from the controller to run admin commands on a compute node.

5.6 Drain watchdog pattern

SLURM auto-clears State=DRAIN on DYNAMIC_NORM nodes when they re-register. To keep a node out of the pool sticky-style, use State=DOWN AND start a watchdog:

# on the controller, as root
nohup bash -c '
  while true; do
    s=$(scontrol show node <FQDN> 2>/dev/null | grep -oE "State=[A-Z+_]+")
    if ! echo "$s" | grep -qE "DOWN|DRAIN"; then
      scontrol update NodeName=<FQDN> State=DOWN Reason="watchdog" >/dev/null 2>&1
    fi
    sleep 10
  done
' > /home/gharunner/_audit/drain_<node>_watchdog.log 2>&1 &

Doesn't survive controller reboots — for permanent removal a SLURM admin should edit slurm.conf.

6. Docker image tag gotchas

Don't invent a "release" tag pattern from a date-suffixed nightly. lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x does not exist — only the dated v0.5.12-rocm720-mi35x-20260517 does. All MI355X sglang-rocm:rocm720 tags follow the dated-nightly pattern.

Before bumping an image, verify the target tag exists:

curl -sI "https://hub.docker.com/v2/repositories/lmsysorg/sglang-rocm/tags/v0.5.12-rocm720-mi35x"
# 200 → exists; 404 → doesn't

Or check whether any other recipe on main uses the proposed tag — if zero uses, suspect.

7. CI: rerun mechanics

  • gh run rerun <id> --failed only works when the workflow run is completed with conclusion=failure. If the run is still queued/in_progress, the call returns "cannot be rerun".
  • To abandon an in-flight run and start fresh, push an empty commit to the PR branch: 
    git commit --allow-empty -m "Re-trigger sweep"
git push
    The old run will be auto-cancelled by workflow/cancel-sweep-on-merge (provided the head SHA changed).
  • For a cancelled run (not failure), use gh run rerun <id> without --failed to re-run everything.

7.1 Reuse after matrix-generation policy changes

Reusable source artifacts are authoritative. The merge-time reuse-ingest-artifacts job validates that downloaded artifacts are readable, non-duplicated, and internally consistent, but it does not require them to match a matrix regenerated from the merge commit. A generator-policy change between the PR sweep and merge therefore does not require another GPU sweep.

Raw and aggregate eval identities must still match, as must agentic point/raw artifacts and summaries. Batched eval identities come from completed_eval_concs, so an explicitly pinned failed run may reuse only the points it completed. Missing or invalid metadata, duplicate identities, and raw/aggregate disagreement still fail reuse.

8. gh CLI gotchas

  • gh pr edit silently aborts on a Projects-classic deprecation GraphQL error. Title/body updates won't apply. Use gh api -X PATCH "repos/<org>/<repo>/pulls/<N>" -f title="..." -F body=@file.md instead.
  • Same issue for adding labels — use gh api -X POST "repos/<org>/<repo>/issues/<N>/labels" -f "labels[]=<name>".
  • gh pr view ... --jq .headRefName output can have a trailing \r. Strip it: gh pr view <N> --json headRefName --jq .headRefName | tr -d '\r\n'. Otherwise shell concatenation produces branchunners/launch_mi300x-amds.sh-style corruption.
  • gh pr list --json statusCheckRollup truncates each PR's rollup — never trust it for per-check filters. Re-query each PR individually with gh pr view <N> --json statusCheckRollup.
  • gh and the GitHub Actions API: conclusion is "" (empty string, not null) for in-flight checks, so jq's // .status fallback doesn't trigger. Use: 
    def state: if (.conclusion // "") != "" then .conclusion else .status end;

9. PR conventions for this repo

  • Image-bump / new-recipe PRs I open on behalf of the user (or that the user creates) get the [Klaud Cold] title prefix.
  • Add the full-sweep-enabled label so a canary-gated full sweep actually runs (gh api -X POST ... labels[]=full-sweep-enabled). Use non-canary-full-sweep-enabled instead only when the single-node canary is flaky or unrepresentative; it runs the full sweep without the canary gate. Without one of the sweep labels, the sweep is mostly SKIPPED.
  • After any code change that shifts a PR's scope (drops a recipe, changes an image tag), update the PR title AND body in the same step and verify with gh pr view <N> --json title,body — gh pr edit silently fails (see §8).
  • utils/merge_with_reuse.sh <N> is the merge entrypoint; it handles the perf-changelog.yaml auto-append.

10. Useful slash commands (defined in .claude/commands/)

  • /find-mergeable-claude-prs — lists claude/* PRs whose full sweep finished all-green.
  • /list-claude-pr-status — lists READY/RUNNING (and optionally FAILED) state per claude/* PR.
  • /fix-klaud-cron-prs — diagnoses failing claude/* PRs by reading their failed job logs.
  • /merge-prs <N> [<N>...] — sequential merge via utils/merge_with_reuse.sh.

Each command file is self-contained; read them to understand the exact jq filters they use.

11. MiniMax M3 B300 MSA top-k slice is non-contiguous

Symptom: MiniMax M3 fails during MSA kernel warmup with:

ValueError: q2k_indices must be contiguous with layout [head_kv, total_q, topK]

The stack ends in sparse_attention_msa.py -> build_k2q_csr(). TP4/TP8 canaries may pass while TP1 data-parallel-attention jobs fail.

Root cause: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 stores top-k indices in a persistent [head_kv, max_num_batched_tokens, topK] buffer for CUDA graphs. The MSA prefill path slices the token dimension before calling build_k2q_csr(). That view retains the full-buffer head stride and is not contiguous when a worker has multiple local KV/index heads. Data-parallel attention forces TP1, exposing all four MiniMax M3 KV/index heads per worker.

Workaround: Before server startup, patch the installed vllm/models/minimax_m3/nvidia/sparse_attention_msa.py assignment from:

prefill_topk = topk[:, nd:num_tokens, :]

to:

prefill_topk = topk[:, nd:num_tokens, :].contiguous()

Use an exact-source guard and remove the workaround once the image includes the fix.

Seen on: #1834.