Finalmask header-custom: add programmable handshake templates and runtime core

[fatyzzz]

Summary

• extend finalmask/header-custom with programmable handshake templates for both TCP and UDP
• add deterministic capture, reuse, and transform fields together with flow-scoped state and runtime-derived metadata
• wire the template model into the existing header-custom runtime while keeping existing packet / rand behavior working

Why

header-custom currently supports fixed bytes and random bytes only. That is enough for static wrappers, but not for handshakes that need captured fields, reused values, deterministic derived fields, or request/response correlation.

This PR keeps the template/config layer and the corresponding runtime core together so the feature is self-contained.

Scope

• expand header-custom config with captured, reused, and transformed fields
• support bounded deterministic transforms: concat, slice, xor16, xor32, be16, and be32
• add flow-scoped state and runtime-derived metadata support for header-custom
• wire the new template semantics into the TCP and UDP header-custom runtime
• keep backward compatibility with existing packet / rand configs

Non-Goals

• no loops
• no branching
• no protocol-specific helpers
• no detached UDP standalone mode in this PR

Test Plan

• go test ./infra/conf -run 'Test(HeaderCustomUDPBuild|HeaderCustomTCPBuildRejectsMixedItemKinds|HeaderCustomUDPBuildRejectsInvalidVariableNames|HeaderCustomUDPBuildRejectsExprWithoutArgs)' -count=1
• go test ./transport/internet/finalmask/header/custom -run 'Test(Evaluator.*|Metadata.*|State.*|DSLTCP.*|DSLUDP.*)' -count=1
• go test ./transport/internet/finalmask -run 'Test(ConnReadWrite|PacketConnReadWrite|TCPcustomStaticHandshakeRoundTrip|TCPcustomClientRejectsMismatchedServerSequence|UDPcustomStaticHeaderWireShape|UDPcustomServerRejectsMismatchedStaticHeader)' -count=1

[fatyzzz]

Thanks. This PR intentionally focuses on the schema/evaluator layer only, so I avoided adding transport-specific runtime examples to keep the review scope narrow.

The corresponding runtime-oriented config usage is split into the follow-up stacked PRs:

• metadata / flow state / runtime integration
• UDP standalone mode for detached handshake packets

If it helps the review, I can also add a small generic header-custom JSON snippet here that demonstrates the new item shapes (save, var, expr, etc.) without tying it to any protocol-specific example.

[Fangliding]

算了这太复杂了 最后加一坨都没人知道怎么用 感觉没有现实意义

[fatyzzz]

算了这太复杂了 最后加一坨都没人知道怎么用 感觉没有现实意义

But Xray-core was never meant to be the simplest solution in the first place.
It has always been a tool with a lot of advanced, engineering-oriented capabilities.

And if we are talking about censorship resistance, then I think this is actually a meaningful step.
The value is precisely that different people can shape their own traffic patterns instead of everyone reusing the same fixed template.

Not every feature has to be for everyone.
In this case, the point is to give capable users more room to build their own behavior, and that itself has practical value in an anti-censorship setting.

[RPRX]

可以接受但设计需要简单通俗些，让大多数人知道该怎么用

[fatyzzz]

可以接受但设计需要简单通俗些，让大多数人知道该怎么用

I think there are two different ways to simplify this, and they have very different trade-offs.

If the concern is mainly about UX and how the feature is presented, then I can simplify the surface without really losing capability:
present it as reusable packet templates with captured/reused fields and a small set of deterministic transforms, instead of making it feel like a mini-language.

But if the expectation is to actually remove syntax/operations from the design itself, then we would start losing generality.
That would make the feature easier to explain, but it would also reduce its usefulness for more varied protocol shapes.

So I want to clarify which direction you mean by “make it simpler”:

1. simpler presentation / simpler user-facing model, while keeping the underlying capability
2. or a real reduction of the syntax and supported operations

Those two lead to very different designs.

[RPRX]

1

刚刚才看到 @iambabyninja 也发邮件说了这个，算是“完全可编程协议”的青春版吧，毕竟那个更复杂我都不确定好不好实现

目前可以先把代理和加密的部分完全交给 VLESS Encryption random，然后 finalmask 做外部流量包装和整形，弄成任何东西

至于 Flow，它在 VLESS Encryption 内部

[fatyzzz]

1

刚刚才看到 @iambabyninja 也发邮件说了这个，算是“完全可编程协议”的青春版吧，毕竟那个更复杂我都不确定好不好实现

目前可以先把代理和加密的部分完全交给 VLESS Encryption random，然后 finalmask 做外部流量包装和整形，弄成任何东西

至于 Flow，它在 VLESS Encryption 内部

Yes, I agree. I’ll think about how to make the design simpler and easier to understand, while keeping the underlying flexibility.

[fatyzzz]

I agree with the concern about making the design easier to understand.

I have reworked the user-facing config surface in this PR to make it read more like programmable handshake packet templates, instead of a mini-language:

• kept the existing packet / rand fields unchanged
• renamed the new field tools to clearer template-oriented terms:
  • save -> capture
  • var -> reuse
  • expr -> transform
• applied the same terminology to nested transform arguments
• updated the focused config tests accordingly

The underlying capability is intentionally unchanged:

• no functionality was removed
• bounded deterministic transforms are still supported
• captured/reused fields are still supported
• the later runtime PRs can still build on the same model

So the goal of this update is to simplify the UX and mental model, without reducing the flexibility needed for programmable handshake shaping.

If you think the public model should be simplified even further, I can continue iterating in that direction.

[LjhAUMEM]

这前后也不一致啊，5919 infra 只是新增 save var expr，到 5920 瞬间变成 capture reuse transform

说实话不是很想来审这种 vibe 啊

另外增加表达式操作感觉不如加个 fallback 有用

[fatyzzz]

说实话不是很想来审这种 vibe 啊

另外增加表达式操作感觉不如加个 fallback 有用

The rename was intentional, not arbitrary.

The earlier feedback was that the design should be simpler and easier to understand, so I adjusted the user-facing terminology to make it read less like a mini-DSL and more like packet-template fields.

I agree that changing naming during review adds churn, so I should stabilize the vocabulary more carefully.

But the goal of the rename was to improve UX, not to change the underlying model.

As for fallback, I agree it is useful, but it solves a different problem.
Fallback helps when matching fails; transforms help construct and correlate handshake fields in the first place.
So I do not think they are interchangeable.

[LjhAUMEM]

The rename was intentional, not arbitrary.

Okay, I noticed that this reuse seed wasn't captured before. What is the expected behavior?

[fatyzzz]

What is the expected behavior?

Good catch.

In the current implementation, unresolved reuse is not rejected at config-build time. It is only rejected later at runtime evaluation, where it fails with an unknown variable error.

So this test was only meant to validate the parser/config shape, not to imply that an uncaptured field can actually be reused successfully.

That said, I agree this is a poor example for a user-facing review context, because it makes the model look more confusing than it should. I’ll adjust the test case to use a properly captured field instead.

Also, this is my first PR in Xray-core, and I’m glad to have the opportunity to communicate with the maintainers and contributors here while iterating on the design.

[Fangliding]

可以接受但设计需要简单通俗些，让大多数人知道该怎么用

我自己搓个格式用标准语言还得写一堆函数和变量呢 用这抽象玩意得多难写啊

[fatyzzz]

用这抽象玩意得多难写啊

Sure, writing this is harder than writing a normal program directly.

But the goal here is not to replace a real programming language.

The goal is to provide a bounded in-config mechanism for programmable handshake shaping inside Xray itself.

So I do not expect it to be “easy” in the same sense as a simple config.
The real question is whether this bounded model is useful enough for the cases it targets.

[RPRX]

尽量不要 vibe 吧，除非人工校正过

[fatyzzz]

尽量不要 vibe 吧，除非人工校正过

yeah, sorry about that

i pushed it once it was working and already usable for me, and only later realized i had not thought enough about the UX side

i’m probably not the pickiest user myself :)

will try to keep it less vibe and more polished from here

[fatyzzz]

I restacked the series and rebased it onto current main.

There were two reasons for this:

1. the previous split between the template/config layer and the runtime/state layer was not the best review boundary
2. the branch had also drifted behind current main

So this PR now represents the self-contained header-custom core:

• template/config surface
• evaluator
• state / metadata
• TCP and UDP runtime integration

The remaining follow-up PR is now only for detached UDP standalone mode.

After the restack, I reran the focused tests and local live smoke checks for both TCP and UDP paths.