RFC: Portable Cache

[jerrykingxyz]

Feature Name: Design Portable Cache
Start Date: 2025/11/14

Summary

Based on Persistent Cache, implement Portable Cache, where the generated cache content supports use within the same project across platforms and paths.

Context

The evolution roadmap of Rspack's caching capabilities is to successively implement memory cache (✅), persistent cache (✅), portable cache (🚧), and remote cache (🗓️).

Implementing Portable Cache requires making the content generated by Persistent Cache platform-independent, enabling different platforms to share the same cache simultaneously. A typical use case is in CI, where Windows, Linux, and macOS can use the same cache for acceleration without generating three separate copies.

Persistent Cache Architecture RFC: #8458

Detailed design

Test

The function of the Portable Cache itself is easy to understand. It primarily affects the content of persistent cache, and we need to add two new test types to ensure its functionality.

Functional Test

The purpose of functional test is mainly to ensure that the basic functions of the portable cache operate normally when it is enabled, with a greater emphasis on the same test item, the same operating system, and different project paths.

Due to the functional requirement for directory change, it is necessary to newly introduce NEXT_MOVE_DIR_START pre-set utility function for CacheCase.

// file.js (config as immutablePath)
export default 1;
// after first shutdown, update content to `export default 2;`
// after second shutdown, update content to `export default 3;`

// index.js
import index from "./file.js"
it("test name", () => {
  if (COMPILER_INDEX == 0) {
    expect(index).toBe(1);
    // Restart the rspack in the current project directory.
    await NEXT_START();
  }
  if (COMPILER_INDEX == 1) {
    // Because file.js is configured as an immutable path, it will not be recompiled even if its content changes.
    expect(index).toBe(1);
    // After shutdown, move the project to another temporary directory and then restart rspack.
    await NEXT_MOVE_DIR_START();
  }
  if (COMPILER_INDEX == 2) {
    // Because caching is portable, the results from file.js can be reused.
    expect(index).toBe(1);
  }
})

Cache Content Test

Functional testing cannot detect some behavioral anomalies in different operating systems, such as to_ne_bytes, etc.

Cache content testing focuses more on the same test item, different operating systems, and different project paths, achieving the testing objective by comparing the contents of .cache files generated by different operating systems.

Since directly comparing the contents of the .cache directory has issues such as uncertain content order, including time fields, etc., we need to customize the comparison logic. The adopted solution is to add a new crate (rspack_storage_compare) to compare the contents of the.cache directory by writing Rust code.

// rspack_storage_compare/main.rs
fn main() {
  let storage1 = Storage::new(path1);
  let storage2 = Storage::new(path2);
  // storage should have same scopes.
  assert_eq!(storage1.getScopes(), storage2.getScopes());
  // compare.
  for scope in storage1.getScopes() {
    let data1 = storage1.load(scope);
    let data2 = storage2.load(scope);
    match scope {
      "snapshot" => {
        // deserialize and compare snapshot
      }
      "make" => {
        // deserialize and compare make 
      }
    }
  }
}

To obtain the .cache files generated by different operating systems, we need to upload the .cache files of cacheCase when the linux and windows tests are completed, and then separately start a task to run the crate for comparison.

Code Implementation

The Portable Cache feature requires an additional switch to be added to the Persistent Cache configuration for control, with the default value set to false.

type ExperimentCacheOptions = {
  type: "memory"
} | {
  type: "persistent",
  buildDependencies?: string[],
  version?: string,
  // add a new field to toggle portable feature, default false.
  portable?: boolean,
  ...
}

Portable Cache is built on top of Persistent Cache, and only requires additional processing during the Serialization/Deserialization of MemoryCache to convert platform-specific data (e.g., convert absolute paths to relative paths).

Portable cache implementation is relatively simple, with mainly three changes

1. Serialization/Deserialization passes through Context config.context and cache.portable related information, which is convenient for corresponding structures to customize the conversion process.

struct Context {
  context_path: PathBuf,
  portable: bool
}

let ctx = Context {
  context_path: config.context,
  portable: config.experiment.cache.portable
};
let bytes = to_bytes(&node, &ctx);
let data = from_bytes(&bytes, &ctx);

2. The value of cache.portable is involved in the calculation of the cache directory, and a new cache should be generated when cache.portable is modified.

3. Some data structures need to implement Serialize / Deserialize separately to ensure the portability of the output. Below are some of the relevant data structures.

ArcPath / PathBuf

For Portable cache, absolute paths must not exist, so we can customize the Serialization/Deserialization logic to convert them into relative paths for storage.

struct ArcPath {}

impl CustomConverter for ArcPath {
  type Target = String;
  fn serialize(&self, ctx: Any) -> Result<Self::Target, SerializeError> {
    let Some(ctx) = ctx.downcast_ref::<CacheableContext>() else {
        return Err(...);
    }
    // get cache.portable from context
    if ctx.portable {
      Ok(self.to_relative_path(ctx.path_context))
    } else {
      Ok(self.to_string())
    }
  }
  fn deserialize(data: Self::Target, ctx: Any) -> Result<Self, DeserializeError> {
    ...
  }
}

User project may use resources outside the project (outside config.context), and the portable cache also needs to convert them into relative directories. According to the current code, if these files do not exist in the new environment, it will trigger the reconstruction of the relevant module, and theoretically, there will be no correctness issues.

ModuleIdentifier

There are also some absolute paths in ModuleIdentifier, which need to be split and replaced during the Serialization/Deserialization process

SourceCode

Absolute paths may appear in SourceCode, which may be deliberately set by the user or injected during loader execution. We will discuss these in two scenarios.

1. Use absolute paths in import

import * from "/root/xxxx";

For this situation, we don't need to replace the path within it; we just need to ensure that the path in the corresponding dependency is replaced.

2. SourceCode contains String or other forms of absolute paths

// code in /root/xxxx
console.log("/root/xxx/xxx")

No replacement is performed in this case, but efforts should be made to avoid cross-platform issues caused by the loader introducing this situation. For rspack internal plugins, it is necessary to ensure that the output does not contain absolute paths, while third-party plugins are constrained by specifications.

[jerrykingxyz]

The RFC currently only lists a few data structures that need to be processed; more data structures will be added to the RFC during implementation.