Skip to content

Commit 5c296f2

Browse files
jieyouxujieyouxu
committed
Add policy for adding ecosystem/integration test jobs/components to rust-lang/rust CI
1 parent ed93e45 commit 5c296f2

File tree

4 files changed

+258
-0
lines changed

4 files changed

+258
-0
lines changed

‎src/SUMMARY.md

+1
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
- [Resources](./compiler/resources.md)
5151
- [Review Policy](./compiler/reviews.md)
5252
- [Proposals, Approval and Stabilization](./compiler/proposals-and-stabilization.md)
53+
- [Adding ecosystem/integration test jobs/components to `rust-lang/rust` CI](./compiler/proposals-and-stabilization/ecosystem-integration-tests.md)
5354
- [Third-party and Out-of-tree Crates Policy](./compiler/third-party-out-of-tree.md)
5455
- [Triage and Prioritization](./compiler/prioritization.md)
5556
- [Operations](./compiler/operations.md)

‎src/compiler/proposals-and-stabilization.md

+6
Original file line numberDiff line numberDiff line change
@@ -329,9 +329,15 @@ circumstance.
329329
- **Approve using:** r+ (compiler leads)
330330
- Open a PR with the change affecting licensing and assign it to the team leads for review
331331

332+
### Adding ecosystem/integration test jobs/components to rust-lang/rust CI
333+
334+
See [*Adding ecosystem/integration test jobs/components to rust-lang/rust CI*](./proposals-and-stabilization/custom-test-jobs.md).
335+
336+
332337
[stabilization_guide]: https://rustc-dev-guide.rust-lang.org/stabilization_guide.html
333338
[tier_policy]: https://doc.rust-lang.org/rustc/target-tier-policy.html
334339
[mcps]: https://github.com/rust-lang/compiler-team/issues?q=label%3Amajor-change
335340
[rfcs]: https://github.com/rust-lang/rfcs
336341
[rust]: https://github.com/rust-lang/rust
337342
[compiler_lint_eg]: https://doc.rust-lang.org/rustc/lints/listing/deny-by-default.html#explicit-builtin-cfgs-in-flags
343+
[ecosystem_testing]: https://rustc-dev-guide.rust-lang.org/tests/ecosystem.html

‎src/compiler/proposals-and-stabilization/ecosystem-integration-tests.md

+248
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Adding ecosystem/integration test jobs/components to rust-lang/rust CI
2+
3+
## Scope
4+
5+
This policy is applicable to proposals for adding new ecosystem and integration test jobs/components
6+
that involve building and testing additional artifacts which may cause [rust-lang/rust] PR CI or
7+
Full CI (the "CI") to fail, or produce a failure message, that may impact other [rust-lang/rust]
8+
contributors who also use the [rust-lang/rust] CI.
9+
10+
For example (non-exhaustive):
11+
12+
- Ecosystem test jobs: Rust for Linux or Fuchsia.
13+
- Integration test components: GCC codegen backend or Cranelift codegen backend.
14+
15+
## Background
16+
17+
[rust-lang/rust] runs a small set of build/test jobs on PR CI (faster, less expensive/exhaustive),
18+
and runs a much larger set of build/test jobs on Full CI. PR CI usually takes around an hour, while
19+
Full CI usually takes around 3 hours. Having ecosystem and integration test jobs/components that:
20+
21+
- may spuriously fail; or
22+
- may genuinely fail but it's not clear who should be consulted about the failure or who is
23+
responsible for fixing the failure; or
24+
- otherwise do not have well documented test job maintainers and failure protocol;
25+
26+
can introduce a lot of friction and frustration for other contributors who now will have to deal
27+
with the failure or understand why the test job failed in the first place, which may now be blocking
28+
their PR. Everyone utilizing the [rust-lang/rust] CI is expected to use it responsibly.
29+
30+
To help with this, please follow the process described below if you would like to propose adding an
31+
ecosystem/integration test job/component to the [rust-lang/rust] CI.
32+
33+
## Process for adding an ecosystem/integration test job/component to rust-lang/rust CI
34+
35+
![Flowchart of the process for adding an ecosystem/integration test job/component to rust-lang/rust
36+
CI](./ecosystem-integration-tests/test-job-flow.svg)
37+
38+
1. Ask the Infrastructure Team on zulip
39+
([#t-infra](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra)) to check if there
40+
would be capacity for the proposed test job/component.
41+
2. Propose using a [Major Change Proposal
42+
(MCP)](../proposals-and-stabilization.md#how-do-i-submit-an-mcp).
43+
- Including the filled out *Ecosystem and Integration Test Job/Component Policy* (see below).
44+
3. Once the MCP is seconded and accepted, create a new issue on [rust-lang/rust] about the proposal,
45+
linking to the MCP, and then nominate the proposal for library team review via `@rustbot label
46+
+I-libs-nominated`.
47+
- Link to the MCP.
48+
4. Once the library team reviews the proposal and has no blocking concerns, submit the
49+
implementation PR to [rust-lang/rust].
50+
- The PR should include an accompanying Ecosystem/Integration Test Job/Component support page in
51+
the in-tree [rustc-dev-guide], ensuring that the *Failure Protocol* is well-documented (see
52+
below). Link to the MCP and issue.
53+
5. The Infrastructure Team will review the implementation PR. Once approved, the
54+
ecosystem/integration test job/component will run in [rust-lang/rust] PR CI or Full Merge CI.
55+
56+
## Ecosystem and Integration Test Job/Component Policy
57+
58+
Please copy and fill out this template (below the separator) as part of the MCP. The policy
59+
questions/explanation themselves will be quoted blocks. Information that the MCP author is expected
60+
to provide are indicated by italicized sentences whose content should be replaced.
61+
62+
NOTE: It is not the intention of this policy to make it frustrating annoying to add an ecosystem
63+
test job/component. We simply wish to gather the necessary background information upfront
64+
(especially working together to figure out a viable *Failure Protocol*) to minimize potential
65+
frustrations from other [rust-lang/rust] contributors if and when the the test job/component do
66+
fail, potentially blocking PR / Full Merge CI in completely unrelated PRs.
67+
68+
---
69+
70+
```markdown
71+
## Ecosystem and Integration Test Job/Component Policy
72+
73+
The ecosystem/integration test job/component ("test job/component") proposed for the
74+
[rust-lang/rust] CI must:
75+
76+
- Be approved by the compiler team through a proposed MCP, where the MCP is seconded by a compiler
77+
team member, and the MCP is accepted with no blocking concerns.
78+
- Have no blocking concerns from the library team.
79+
- Have the implementation PR be reviewed and approved by the infrastructure team.
80+
- Be properly documented on [rustc-dev-guide] (preferably as part of the implementation PR).
81+
82+
Please complete the sections below so [rust-lang/rust] teams can have sufficient context about the
83+
proposed test job/component.
84+
85+
### Test job/component rationale
86+
87+
> What does this test job/component do?
88+
>
89+
> - If an ecosystem test job/component is being proposed, can you briefly describe the intended
90+
> ecosystem users?
91+
92+
*Please provide responses here, replacing this sentence.*
93+
94+
> What [rust-lang/rust] changes can potentially break the test job/component?
95+
>
96+
> E.g. changes to rustc, standard library, bootstrap or tools (like clippy/rustfmt/cargo).
97+
98+
*Please provide responses here, replacing this sentence.*
99+
100+
> Why does this test job/component need to be part of the [rust-lang/rust] PR and/or Full Merge CI?
101+
102+
*Please provide responses here, replacing this sentence.*
103+
104+
> If the test job/component will block on failure, why does it need to block?
105+
106+
*Please provide responses here, replacing this sentence.*
107+
108+
> If the test job/component will not block on failure initially but is intended to eventually become
109+
> blocking:
110+
>
111+
> - Why will it become blocking?
112+
> - When will it become blocking?
113+
114+
*Please provide responses here, replacing this sentence.*
115+
116+
### Test job/component maintainers
117+
118+
> The proposed test job/component for [rust-lang/rust] CI must have at least one dedicated test
119+
> job/component maintainer. The test job/component maintainers understand that they will be pinged
120+
> or otherwise contacted about the ecosystem/integration test job/component, particularly for (but
121+
> not limited to) its failures.
122+
>
123+
> **Please list who will be maintaining this ecosystem/integration test job/component here. Please
124+
> format the github handles in the style:**
125+
>
126+
> ```
127+
> [@github_handle_1](https://github.com/github_handle_2)
128+
> [@github_handle_2](https://github.com/github_handle_2)
129+
> ```
130+
>
131+
> NOTE: For future readers, you can paste the usernames without formatting them as links via
132+
> **ctrl-shift-v**.
133+
134+
*Please list test job/component maintainers here with the formatting advice above, replacing this
135+
sentence.*
136+
137+
> **NOTE: If an ecosystem/integration test job/component no longer has an active dedicated
138+
> maintainer (or maintainers), and if [rust-lang/rust] teams find the ecosystem/integration test
139+
> job/component causes significant burden or becomes irrelevant, then the ecosystem/integration test
140+
> job/component may be removed.**
141+
142+
### CI infrastructure considerations
143+
144+
> You should ask the Infrastructure Team on the
145+
> [`#t-infra`](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra) zulip channel when
146+
> proposing a new ecosystem/integration test job/component to check if there's capacity for the test
147+
> job/component.
148+
>
149+
> - Does the ecosystem/integration test job/component require substantial CI resources (storage and/
150+
> or CI time)? In particular, will it require large runners?
151+
152+
*Please provide responses here, replacing this sentence. If there is a zulip topic discussing it
153+
with the Infrastructure Team, please include the zulip topic link here.*
154+
155+
### Features and implementation details
156+
157+
> Does the proposed test job/component intend to use any unstable features?
158+
>
159+
> - If so, are the unstable features ready for exposure (e.g. must an unstable feature be completely
160+
> reworked)?
161+
> - For ecosystem test jobs/components, are the unstable features ready for such exposure to the
162+
> ecosystem, and are the feature stakeholders ready for such usage?
163+
164+
*Please provide responses here, replacing this sentence.*
165+
166+
> Does the proposed test job/component intend to intentionally depend on any implementation details?
167+
> This may include but is not limited to: unstable/internal compiler/tool flags and behaviors,
168+
> `RUSTC_BOOTSTRAP` usages, standard library implementation details, etc.
169+
>
170+
> - If so, are there plans to shrink or expand such dependencies in the future?
171+
172+
*Please provide responses here, replacing this sentence.*
173+
174+
### Failure protocol: what to do if the job/component breaks/fails?
175+
176+
> **NOTE: If the artifacts of an ecosystem/integration test job/component are not shipped as part of
177+
> a distribution component/toolchain, the test job/component may be temporarily disabled to unblock
178+
> [rust-lang/rust] PR CI or Full Merge CI without receiving prior approval from the test
179+
> job/component maintainers. The test job/component maintainers will be pinged or otherwise notified
180+
> about the test job/component being disabled.**
181+
182+
> How can the test job/component maintainers be contacted in case of failure? By default, it is
183+
> assumed that the test job/component maintainer can be pinged via their GitHub handles.
184+
185+
*Please provide responses here, replacing this sentence.*
186+
187+
> (If applicable) If the addition of an ecosystem/integration test job is being proposed:
188+
>
189+
> - How can the test job be run in CI? If so, is there a try job (`try-job: ...`) invocation? What's
190+
> the job name?
191+
> - Can the test job be run locally? If so, how?
192+
193+
*If applicable, please provide responses here, replacing this sentence. Otherwise, you can ignore
194+
this question.*
195+
196+
> (If applicable) If the addition of an ecosystem/integration test component is being proposed:
197+
>
198+
> - Which existing CI jobs will be building and testing this test component?
199+
> - Can they be built and ran as part of a try job? If so, what are the job names and the try job
200+
> (`try-job: ...`) invocations?
201+
> - Can the test component be built and run locally? If so, how?
202+
203+
*If applicable, please provide responses here, replacing this sentence. Otherwise, you can ignore
204+
this question.*
205+
206+
> How can the test job/component be disabled in the event of spurious failures that are blocking PR
207+
> and/or Full Merge CI?
208+
209+
*Please provide responses here, replacing this sentence.*
210+
211+
> If a PR breaks the test job/component:
212+
>
213+
> - If the breakage seems **spurious** and retrying does not resolve the spurious breakage, the test
214+
> job may be **temporarily disabled** (see below).
215+
> - If the breakage is **intentional**, how will this be resolved?
216+
> - If the breakage is **unintentional**, is the PR author expected to fix the breakage?
217+
218+
*Please provide responses here, replacing this sentence.*
219+
220+
### Dependencies, build/test environments and reliability
221+
222+
> Does the test job/component involve any custom build systems that are not used in the regular
223+
> [rust-lang/rust] CI jobs?
224+
225+
*Please provide responses here, replacing this sentence.*
226+
227+
> Does the test job/component depend on external resources (e.g. external servers) that may be
228+
> subject to network connectivity?
229+
>
230+
> - If so, does the infrastructure team need to help maintain a mirror of the required assets?
231+
232+
*Please provide responses here, replacing this sentence.*
233+
234+
> Are there any potential sources of spurious failures due to the test job/component?
235+
236+
*Please provide responses here, replacing this sentence.*
237+
238+
> Are there any other unusual requirements (build environment, dependencies, etc.)?
239+
240+
*Please provide responses here, replacing this sentence.*
241+
242+
243+
[rust-lang/rust]: https://github.com/rust-lang/rust
244+
[rustc-dev-guide]: https://github.com/rust-lang/rustc-dev-guide
245+
```
246+
247+
[rust-lang/rust]: https://github.com/rust-lang/rust
248+
[rustc-dev-guide]: https://github.com/rust-lang/rustc-dev-guide

‎src/compiler/proposals-and-stabilization/ecosystem-integration-tests/test-job-flow.svg

+3
Loading

0 commit comments

Comments
 (0)