Shadow support

[xDimon]

Description

This PR introduces the x-shadow feature flag that enables building binaries compatible with the Shadow network simulation environment. When this feature is enabled, several runtime and I/O adjustments are applied to ensure the Polkadot binaries can run correctly within simulated network conditions.

Summary of changes

• Allocator: Disabled jemalloc, as its internal locking can lead to deadlocks in Shadow simulations.
• IPC Mechanism: Replaced inter-process communication between the relay node and PVF workers from Unix domain sockets (UDS) to TCP sockets, since UDS are not supported by Shadow.
• Security Sandbox: Fallback to the default security mode because some system calls (e.g. madvise) required for fine-grained sandboxing are unavailable in Shadow.
• CPU Usage Check: Disabled CPU time consumption checks for PVF workers, since the required getrusage syscall is not supported in Shadow.
• Litep2p Compatibility: Updated code to maintain compatibility with the latest version of litep2p, which includes internal fixes and API adjustments (Add x-shadow feature for Shadow simulation compatibility litep2p#446)
• Comments: Improved inline comments and clarified feature-specific code paths.

The result is a cleanly switchable configuration via the x-shadow feature flag, allowing builds for both real-world and simulated environments without code duplication.

Integration

When building for Shadow simulations, enable the feature flag:

cargo build --features x-shadow --release -p polkadot

No external configuration changes are required for downstream projects unless they depend directly on the modified worker interfaces. Existing builds and runtime behavior remain unchanged when the feature is disabled.

Integration testing was performed in a Shadow simulation with multiple configurations of:

• Number of validators,
• Number of parachains,
• Number of collators.

The following setups were used:

• Relay chain: rococo-local specification, binary polkadot
• Parachains: asset-hub-rococo-local specification, binary polkadot-parachain

All tested combinations ran successfully and stably in the simulated environment.

Review Notes

Implementation focuses on compatibility and isolation of Shadow-specific logic using conditional compilation via the x-shadow feature flag. This ensures minimal divergence from the mainline codebase.

Key technical changes include:

• Replacing UnixStream/UnixListener with TcpStream/TcpListener in PVF worker communication.
• Wrapping feature-dependent code with #[cfg(feature = "x-shadow")].
• Adjusting worker safety checks and resource accounting under simulation constraints.
• Maintaining parity with upstream functionality for non-Shadow builds.

No new dependencies introduced. Manual simulation testing performed under Shadow confirmed expected runtime behavior.

[s0me0ne-unkn0wn]

Unrelated?

[xDimon]

  error: `#[deprecated]` attribute cannot be used on function params
     --> /Users/di/Projects/polkadot/substrate/frame/executive/src/lib.rs:205:2
      |
  205 | /     #[deprecated(
  206 | |         note = "`OnRuntimeUpgrade` parameter in Executive is deprecated, will be removed after September 2026. \
  207 | |         Use type `SingleBlockMigrations` in frame_system::Config instead."
  208 | |     )] OnRuntimeUpgrade = (),
      | |______^
      |
      = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
      = help: `#[deprecated]` can be applied to functions, data types, modules, unions, constants, statics, macro defs, type aliases, use statements, foreign statics, struct fields, traits, associated types, associated consts, enum variants, inherent impl blocks, and crates
      = note: `#[deny(useless_deprecated)]` on by default

That's why I deleted it. Anyway, this message had already been left in the doc-comment.

[s0me0ne-unkn0wn]

Just briefly skimmed it for now. Main though is, given that you don't need resource counters, process isolation, and sandboxing per se, why do you want to complicate it with TCP sockets at all? Wouldn't it be more straightforward to get rid of workers altogether and execute PVFs inside the candidate validation subsystem?

[xDimon]

Wouldn't it be more straightforward to get rid of workers altogether and execute PVFs inside the candidate validation subsystem?

It would be reasonable. Could you give some hint how to do it?

[s0me0ne-unkn0wn]

It would be reasonable. Could you give some hint how to do it?

I'd create a hacked version of handle_execute_pvf which does not pass it down the pipeline but just prepares and executes the PVF in place instead (checks if the artifact is in cache, calls executor_interface::prepare if it's not, then calls executor_interface::execute_artifact). handle_precheck_pvf may be hacked to either always return successful pre-checking result or to do the pre-checking the similar blocking way. That may or may not be sufficient depending on what you're testing. If you're running a single parachain, I'm pretty much sure it's enough. If you're running multiple parachains and care about latency, you may want a little bit more sophisticated implementation, but still, a pretty simple one (like, you may spawn a thread when you need to prepare an artifact, and then execute it as soon as the thread joins).