feat: add distributed owner SSD backing tier for KV cache storage

.github/workflows/all_test.yml

@@ -54,6 +54,8 @@ jobs:
             protobuf-compiler \
             python3-venv \
             rsync
+          sudo apt-get clean
+          sudo rm -rf /var/lib/apt/lists/*
 
       - name: Verify Docker availability
         run: |
@@ -87,6 +89,7 @@ jobs:
               --skip-dispatch \
               --skip-start-testbed \
               --skip-runner \
+              --cleanup-pack-runtime-after-success \
               --print-generated
 
       - name: Archive Fluxon CI release
@@ -214,6 +217,7 @@ jobs:
               --suite-path "$RUNNER_TEMP/ci_test_list.ci.yaml" \
               --print-generated \
               --testbed-hostworkdir "$RUNNER_TEMP/fluxon_deploy" \
+              --cleanup-successful-case-artifacts \
               --skip-pack
           rc=$?
           if [ "$rc" -eq 0 ]; then
@@ -225,7 +229,7 @@ jobs:
           exit "$rc"
 
       - name: Normalize ci_2_virt_node debug artifact permissions
-        if: ${{ always() }}
+        if: ${{ failure() }}
         run: |
           chmod -R a+rX .dever/ci_2_virt_node || true
           find .dever/ci_2_virt_node -type f -name '*.yaml' -exec chmod a+r {} + || true
@@ -258,7 +262,7 @@ jobs:
           path: ${{ runner.temp }}/codex-failure-context/**
 
       - name: Upload ci_2_virt_node debug artifacts
-        if: ${{ always() }}
+        if: ${{ failure() }}
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: ci-2-virt-node-debug-${{ github.sha }}

AGENTS.md

@@ -1,5 +1,8 @@
 Keep this document concise.
 - Core user, developer, and design docs are in-repo under fluxon_doc_cn/ and fluxon_doc_en/
+- Detailed bilingual doc writing rules are indexed at `fluxon_doc_en/dev_doc/Developer - 3 - Documentation Writing Rules.md` and `fluxon_doc_cn/dev_doc/开发者 - 3 - 文档写作规约.md`
+- The bilingual technical copy-editing workflow and one-shot example are at `fluxon_doc_en/dev_doc/Developer - 5 - Technical Documentation Copy Editing.md` and `fluxon_doc_cn/dev_doc/开发者 - 5 - 技术文档审校.md`
+- Bilingual code review rules, finding levels, and review templates are at `fluxon_doc_en/dev_doc/Developer - 6 - Code Review Guidelines.md` and `fluxon_doc_cn/dev_doc/开发者 - 6 - Code Review 规约.md`
 - Repository-level rules and component contracts are indexed under Contract Index below. Keep AGENTS summaries concise; put reusable details in linked bilingual docs and update both language indexes together.
 - teststack has two steps: start testbed and testrunner
 - teststack has UI support; testrunner should own the UI authority and API surface, and the UI should run as a long-lived service that reuses the ops interfaces underneath
@@ -9,6 +12,9 @@ Keep this document concise.
 - Git operations are limited to basic `stage`, `unstage`, `commit`, and `push`. Do not use other Git operations.
 - Prefer contraction over compatibility by default. Do not add compatibility layers, deprecated paths, or aliases unless the task explicitly requires them.
 - Prefer one canonical name for one concept. Avoid synonym parameters, duplicated entrypoints, and parallel config surfaces.
+- When a change crosses module boundaries or moves resource cleanup, identify one final-release owner. Keep public layers on contracts, composition layers on ordering, and internal modules on their own state and cleanup; fix the invariant at its owner without adding outer field mutation or a duplicate close path.
+- When a change adds lifecycle dependencies or background work, write the dependency order and implement shutdown as admission stop, scoped wake or cancel, quiescence, dependent release, then dependency release. Keep independent branches parallel and avoid a global lock or coordinator unless the dependency graph requires it.
+- When shutdown intent and cleanup completion can diverge, assign each state a scope and sole writer, keep transitions monotonic, make close repeatable, and treat successful close as a completion barrier. Do not add multiple sources of truth for the same lifecycle fact.
 - Do not add forwarding wrappers that only rename a call, pass through arguments, or unpack/repack its result without adding a contract, validation, transformation, or ownership boundary. Call the canonical implementation directly.
 - Do not use environment variables for ordinary parameter passing. Prefer configuration files first, then explicit command-line arguments.
 - Prefer convention over configuration. When one canonical path or default wiring is sufficient, do not add extra config knobs.

AGENTS_CN.md

@@ -1,5 +1,8 @@
 保持本文档简洁。
 - 核心用户文档、开发文档和设计文档都在仓库内的 `fluxon_doc_cn/` 和 `fluxon_doc_en/` 下
+- 详细的中英文文档写作规约索引见 `fluxon_doc_cn/dev_doc/开发者 - 3 - 文档写作规约.md` 和 `fluxon_doc_en/dev_doc/Developer - 3 - Documentation Writing Rules.md`
+- 中英文技术文档审校流程和 one-shot 示例见 `fluxon_doc_cn/dev_doc/开发者 - 5 - 技术文档审校.md` 和 `fluxon_doc_en/dev_doc/Developer - 5 - Technical Documentation Copy Editing.md`
+- 中英文 Code Review 规约、finding 等级和 Review 模板见 `fluxon_doc_cn/dev_doc/开发者 - 6 - Code Review 规约.md` 和 `fluxon_doc_en/dev_doc/Developer - 6 - Code Review Guidelines.md`
 - 仓库级规则和组件契约统一收录在下方的“规约索引”中。`AGENTS` 只保留简明摘要；可复用细则写入索引指向的双语文档，并同步更新中英文索引。
 - `teststack` 有两个步骤：`start testbed` 和 `testrunner`
 - `teststack` 支持 UI；`testrunner` 应负责 UI 的 authority 和 API surface，但 UI 应作为常驻服务运行，并复用下层的 ops 接口
@@ -9,6 +12,9 @@
 - Git 操作仅限基础的 `stage`、`unstage`、`commit` 和 `push`。不要使用其他 Git 操作
 - 默认优先收束而不是兼容。除非任务明确要求，否则不要添加兼容层、废弃路径或别名
 - 一个概念优先只保留一个正式名字。避免同义参数、重复入口和并行配置面
+- 当改动跨越模块边界或移动资源清理职责时，必须指定唯一 final-release owner。公共层只依赖契约，组合层只安排顺序，内部模块维护自己的状态和 cleanup；在不变量 owner 处修复，不增加外层字段操作或第二条 close 路径
+- 当改动新增生命周期依赖或后台任务时，必须写明依赖顺序，并按“关闭入口、定向 wake/cancel、建立静默点、释放依赖方、最后释放被依赖方”执行关闭。互不依赖的分支保持并行，依赖图不需要时不引入全局锁或全局 coordinator
+- 当停止意图和清理完成可能分离时，必须为每个状态指定作用域和唯一 writer，保持状态单调、close 可重复，并把 close 成功作为完成屏障。不要为同一个生命周期事实增加多个 source of truth
 - 不要添加只做调用改名、参数透传或结果拆包 / 重包，却不增加契约、校验、转换或所有权边界的转发包装；应直接调用唯一的标准实现
 - 不要用环境变量传递普通参数。优先使用配置文件，其次使用显式命令行参数
 - 优先约定而不是配置。当一个标准路径或默认连接方式已经足够时，不要增加额外配置项

README.md

@@ -134,6 +134,12 @@ The `TCP Benchmark` shows that Fluxon outperforms `MooncakeStore` and `Redis` on
 
 ![](./pics/kv_benchmark_chart.png)
 
+Fluxon KV can also use an owner's local SSD as a runtime backing layer for DRAM replicas. The chart below shows a single-node H100 SSD-pressure experiment measured through CUDA event completion. At c16, Fluxon's hit payload throughput for 4/8/16 MiB payloads was 3.83×/5.61×/6.73× that of the faster of the two Mooncake topologies.
+
+![GPU-ready KV throughput and hit rate across Fluxon and Mooncake SSD configurations](./pics/kv_ssd_gpu_sweep.png)
+
+See the Chinese deep dive, [Fluxon KV SSD Storage: Using Local SSD as a Backing Layer for In-Memory Replicas](https://tele-ai.github.io/Fluxon/cn/blog/blog_2_Fluxon_KV_SSD%E5%AD%98%E5%82%A8%EF%BC%9A%E6%8A%8A%E6%9C%AC%E5%9C%B0SSD%E6%8E%A5%E6%88%90%E5%86%85%E5%AD%98%E5%89%AF%E6%9C%AC%E7%9A%84%E5%9B%9E%E5%A1%AB%E5%B1%82), for the full setup, hit rates, and scope.
+
 ### Fluxon FS Benchmark
 
 The benchmark results show that small-file reads and large-file writes already outperform `Alluxio`, large-file read performance is broadly on par, and small-file write performance still has further room to improve.
@@ -294,6 +300,9 @@ Contributions are welcome. Before you start, please read the developer docs on G
 - [Developer - 2 - Package middleware and images](https://tele-ai.github.io/Fluxon/dev_doc/Developer---2---Package-Middleware-and-Images/)
 - [Developer - 3 - Documentation Writing Rules](https://tele-ai.github.io/Fluxon/dev_doc/Developer---3---Documentation-Writing-Rules/)
 - [Developer - 4 - Publish a release](https://tele-ai.github.io/Fluxon/dev_doc/Developer---4---Publish-a-Release/)
+- [Developer - 5 - Technical Documentation Copy Editing](https://tele-ai.github.io/Fluxon/dev_doc/Developer---5---Technical-Documentation-Copy-Editing/)
+- [Developer - 6 - Code Review Guidelines](https://tele-ai.github.io/Fluxon/dev_doc/Developer---6---Code-Review-Guidelines/)
+- [Developer - 7 - Event Subscription and Full Snapshot Guidelines](https://tele-ai.github.io/Fluxon/dev_doc/Developer---7---Event-Subscription-and-Full-Snapshot-Guidelines/)
 
 <a id="contributors"></a>
 

README_CN.md

@@ -136,6 +136,12 @@ Fluxon FS 是一款面向 AI 数据与模型文件、兼容 `S3` 的高性能文
 
 ![](./pics/kv_benchmark_chart.png)
 
+Fluxon KV 还可将 owner 本地 SSD 作为 DRAM 副本的运行期回填层。下图为单机 H100 的 SSD-pressure 实验，以 CUDA event 完成为计数边界；c16 下，Fluxon 在 4/8/16 MiB 组中的 hit payload 吞吐相对两种 Mooncake 拓扑中更快的一种分别达到 3.83×/5.61×/6.73×。
+
+![Fluxon 与 Mooncake 不同 SSD 配置下的 GPU 可用 KV 吞吐和命中率](./pics/kv_ssd_gpu_sweep.png)
+
+完整实验设置、命中率和适用边界见博客：[Fluxon KV SSD 存储：把本地 SSD 接成内存副本的回填层](https://tele-ai.github.io/Fluxon/cn/blog/blog_2_Fluxon_KV_SSD%E5%AD%98%E5%82%A8%EF%BC%9A%E6%8A%8A%E6%9C%AC%E5%9C%B0SSD%E6%8E%A5%E6%88%90%E5%86%85%E5%AD%98%E5%89%AF%E6%9C%AC%E7%9A%84%E5%9B%9E%E5%A1%AB%E5%B1%82)。
+
 ### Fluxon FS 基准测试
 
 测试结果显示，小文件读取和大文件写入性能已显著优于 `Alluxio`，大文件读取性能基本持平，小文件写入性能仍有进一步优化的空间。
@@ -296,6 +302,9 @@ ui
 - [开发者 - 2 - 打包中间件和镜像](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---2---%E6%89%93%E5%8C%85%E4%B8%AD%E9%97%B4%E4%BB%B6%E5%92%8C%E9%95%9C%E5%83%8F/)
 - [开发者 - 3 - 文档写作规约](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---3---%E6%96%87%E6%A1%A3%E5%86%99%E4%BD%9C%E8%A7%84%E7%BA%A6/)
 - [开发者 - 4 - 发布 Release](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---4---%E5%8F%91%E5%B8%83-Release/)
+- [开发者 - 5 - 技术文档审校](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---5---%E6%8A%80%E6%9C%AF%E6%96%87%E6%A1%A3%E5%AE%A1%E6%A0%A1/)
+- [开发者 - 6 - Code Review 规约](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---6---Code-Review-%E8%A7%84%E7%BA%A6/)
+- [开发者 - 7 - 事件订阅与全量快照规约](https://tele-ai.github.io/Fluxon/cn/dev_doc/%E5%BC%80%E5%8F%91%E8%80%85---7---%E4%BA%8B%E4%BB%B6%E8%AE%A2%E9%98%85%E4%B8%8E%E5%85%A8%E9%87%8F%E5%BF%AB%E7%85%A7%E8%A7%84%E7%BA%A6/)
 
 <a id="contributors"></a>
 

deployment/gen_k8s_daemonset.py

@@ -349,7 +349,7 @@ def _mirror_daemonset_yaml_outdir(*, src_outdir: str, dst_outdir: str) -> None:
     #
     # Causal chain:
     # - Operators often run the generator from the repo workdir, while fluxon-deployer reads
-    #   manifests from a hostworkdir-mounted directory (e.g. /opt/store_team_dev/fluxon_deployer).
+    #   manifests from a hostworkdir-mounted directory (e.g. /tmp/fluxon-example/deploy).
     # - If the two directories drift, deployer applies stale manifests and can crash-loop.
     # - Therefore we provide an explicit opt-in "mirror_outdir" that:
     #   - cleans the destination directory (same semantics as the primary outdir)

deployment/tests/test_build_doc_site_contract.py

@@ -99,6 +99,16 @@ def test_rewrite_homepage_target_path_for_chinese_home(self) -> None:
             "../fluxon_rs/rust-toolchain.toml",
         )
 
+    def test_rewrite_variant_repo_image_for_quartz_route(self) -> None:
+        for language in ("en", "cn"):
+            with self.subTest(language=language):
+                self.assertEqual(
+                    _DOC_SITE.rewrite_variant_target_path(
+                        "../../pics/kv_ssd_cpu_sweep.png", language=language
+                    ),
+                    "../pics/kv_ssd_cpu_sweep.png",
+                )
+
     def test_stage_cn_only_design_into_en_root_when_english_design_missing(self) -> None:
         tmpdir = Path(tempfile.mkdtemp(prefix="fluxon_doc_site_contract_"))
         old_stage_root = _DOC_SITE.STAGE_DOCS_ROOT

deployment/tests/test_start_test_bed_bootstrap_log.py

@@ -1026,11 +1026,11 @@ def test_bare_then_apply_success_path_does_not_run_post_apply_stop() -> None:
   gitops_config_path: ./gitops.yaml
 bootstrap_phases:
   - mode: fixed_bare
-    node: infra44-ThinkStation-PX
+    node: example-node-a
     services:
       - etcd
   - mode: fixed_bare
-    node: infra44-ThinkStation-PX
+    node: example-node-a
     services:
       - fluxon_core_controller
 deploy_workloads:
@@ -1052,7 +1052,7 @@ def test_bare_then_apply_success_path_does_not_run_post_apply_stop() -> None:
   FLUXON_RELEASE_PYLIB_SRC_TAR: pylib_src.tar.gz
   FLUXON_RELEASE_WHEEL: fluxon-0.2.1-cp38-abi3-manylinux_2_28_x86_64.whl
 cluster_nodes:
-  - hostname: infra44-ThinkStation-PX
+  - hostname: example-node-a
     ip: 127.0.0.1
     hostworkdir: /tmp/fluxon_testbed
     ssh_user: tester
@@ -1061,28 +1061,28 @@ def test_bare_then_apply_success_path_does_not_run_post_apply_stop() -> None:
 atomic_groups:
   fluxon_core_controller:
     phase: 1
-    nodes: ["infra44-ThinkStation-PX"]
+    nodes: ["example-node-a"]
     services: ["master", "owner", "ops_controller", "ops_agent"]
 bootstrap_bare_services: ["etcd"]
 service:
   etcd:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   master:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   owner:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   ops_controller:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   ops_agent:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   fluxon_fs_agent:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
 """.strip()
             + "\n",
             encoding="utf-8",
@@ -1257,11 +1257,11 @@ def test_bare_only_stops_after_controller_ready() -> None:
   gitops_config_path: ./gitops.yaml
 bootstrap_phases:
   - mode: fixed_bare
-    node: infra44-ThinkStation-PX
+    node: example-node-a
     services:
       - etcd
   - mode: fixed_bare
-    node: infra44-ThinkStation-PX
+    node: example-node-a
     services:
       - fluxon_core_controller
   - mode: coverage_bare
@@ -1287,7 +1287,7 @@ def test_bare_only_stops_after_controller_ready() -> None:
   FLUXON_RELEASE_PYLIB_SRC_TAR: pylib_src.tar.gz
   FLUXON_RELEASE_WHEEL: fluxon-0.2.1-cp38-abi3-manylinux_2_28_x86_64.whl
 cluster_nodes:
-  - hostname: infra44-ThinkStation-PX
+  - hostname: example-node-a
     ip: 127.0.0.1
     hostworkdir: /tmp/fluxon_testbed
     ssh_user: tester
@@ -1296,31 +1296,31 @@ def test_bare_only_stops_after_controller_ready() -> None:
 atomic_groups:
   fluxon_core_controller:
     phase: 1
-    nodes: ["infra44-ThinkStation-PX"]
+    nodes: ["example-node-a"]
     services: ["master", "owner", "ops_controller", "ops_agent"]
 bootstrap_bare_services: ["etcd"]
 service:
   etcd:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   greptime:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   master:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   owner:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   ops_controller:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   ops_agent:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
   fluxon_fs_agent:
     node_bind:
-      node: ["infra44-ThinkStation-PX"]
+      node: ["example-node-a"]
 """.strip()
             + "\n",
             encoding="utf-8",
@@ -1419,15 +1419,15 @@ def _fail_deploy_or_wait(**_: object) -> dict[str, str]:
             {
                 "launches": [
                     {
-                        "node": "infra44-ThinkStation-PX",
+                        "node": "example-node-a",
                         "selection_name": "etcd",
                     }
                 ]
             },
             {
                 "launches": [
                     {
-                        "node": "infra44-ThinkStation-PX",
+                        "node": "example-node-a",
                         "selection_name": "fluxon_core_controller",
                     }
                 ]
@@ -1445,7 +1445,7 @@ def test_initial_controller_handover_uses_local_bare_stop() -> None:
     with tempfile.TemporaryDirectory(prefix="test_start_test_bed_controller_handover_stop_") as td:
         workdir = Path(td)
         local_node_cfg = {
-            "hostname": "infra44-ThinkStation-PX-a",
+            "hostname": "example-node-a-a",
             "ip": "127.0.0.1",
             "hostworkdir": str(workdir / "hostworkdir"),
             "execution_mode": "local",
@@ -1457,15 +1457,15 @@ def test_initial_controller_handover_uses_local_bare_stop() -> None:
             "atomic_groups": {
                 "fluxon_core_controller": {
                     "phase": 1,
-                    "nodes": ["infra44-ThinkStation-PX-a"],
+                    "nodes": ["example-node-a-a"],
                     "services": ["master", "owner", "ops_controller", "ops_agent"],
                 }
             },
             "service": {
-                "master": {"node_bind": {"node": ["infra44-ThinkStation-PX-a"]}},
-                "owner": {"node_bind": {"node": ["infra44-ThinkStation-PX-a"]}},
-                "ops_controller": {"node_bind": {"node": ["infra44-ThinkStation-PX-a"]}},
-                "ops_agent": {"node_bind": {"node": ["infra44-ThinkStation-PX-a"]}},
+                "master": {"node_bind": {"node": ["example-node-a-a"]}},
+                "owner": {"node_bind": {"node": ["example-node-a-a"]}},
+                "ops_controller": {"node_bind": {"node": ["example-node-a-a"]}},
+                "ops_agent": {"node_bind": {"node": ["example-node-a-a"]}},
             },
         }
         stop_calls: list[str] = []
@@ -1485,10 +1485,10 @@ def test_initial_controller_handover_uses_local_bare_stop() -> None:
             module._run_local_stop = original_run_local_stop
         assert stopped == ["master", "owner", "ops_controller", "ops_agent"], stopped
         assert stop_calls == [
-            "infra44-ThinkStation-PX-a:master",
-            "infra44-ThinkStation-PX-a:owner",
-            "infra44-ThinkStation-PX-a:ops_controller",
-            "infra44-ThinkStation-PX-a:ops_agent",
+            "example-node-a-a:master",
+            "example-node-a-a:owner",
+            "example-node-a-a:ops_controller",
+            "example-node-a-a:ops_agent",
         ], stop_calls
     print("PASS: test_initial_controller_handover_uses_local_bare_stop")