Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CI caching #9552

Merged
merged 1 commit into from
Feb 29, 2024
Merged

Update CI caching #9552

merged 1 commit into from
Feb 29, 2024

Conversation

MonicaOlejniczak
Copy link
Contributor

@MonicaOlejniczak MonicaOlejniczak commented Feb 29, 2024

↪️ Pull Request

These changes update the CI workflow Rust caching so that it can be better reused across jobs, halving the storage used from 1849MB → 920MB. The specific changes include:

  • Remove linting cache as it takes the same amount of time as without a cache, and only wastes a bit of space given the 10GB limit in Github
  • Remove unused components: rustfmt installation from test jobs
  • Reuse the cache between integration and unit tests by adding a shared-key — this overrides the default per job cache and is more effective since we are using the same set of OS + targets for both jobs
  • Update workflows to make them consistent by placing name at the top and using ${{ }} over ${{}} for expressions
  • Use runner.os over matrix.os as it is always available

💻 Examples

Observe the CI workflow behaviour, given the following example:

  • Linting still takes 30s without a cache, the same as this branch that has caching
  • Same cache is found for unit and integration tests across node versions
    • v0-rust-ubuntu-latest-a554fcb7-c1764bc1
    • v0-rust-macos-latest-4be4b021-c1764bc1
    • v0-rust-windows-latest-569c1dff-c1764bc1
  • Linux step only runs on ubuntu jobs
  • Build native release takes ~1-3min, the same as the branch here
  • Observe caches here which are ~200-250MB for each target
  • Missed cache takes ~2x as long to build native release compared to cache hit

Note: REPL fails here because it is tested on push not pull request, and windows has been consistently failing on main branch

🚨 Test instructions

Look at the CI step for this pull request, or otherwise test on your own branch by adding a push trigger

Copy link
Contributor

@mattcompiles mattcompiles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one

Copy link
Contributor

@marcins marcins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it, thanks!

@@ -79,29 +80,30 @@ jobs:
packages/core/rust/*.node

integration_tests:
name: Integration tests (${{matrix.os}}, Node ${{matrix.node}})
name: Integration tests (${{ matrix.os }}, Node ${{ matrix.node }})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these whitespace changes just some auto-formatting somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not auto-formatting, it was done manually and matches other workflow files. It's the style the Github documentation uses everywhere.

strategy:
matrix:
node: [18, 20]
os: [ubuntu-latest, macos-latest, windows-latest]
# These tend to be quite flakey, so one failed instance shouldn't stop
# others from potentially succeeding
fail-fast: false
runs-on: ${{matrix.os}}
runs-on: ${{ matrix.os}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/nit this whitespace change seems inconsistent - no trailing space.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.. related, maybe we should have prettier run on these .yml files? I think that's just adding it to

"*.{js,json,md,ts}": "prettier --write",

Copy link
Contributor Author

@MonicaOlejniczak MonicaOlejniczak Feb 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I missed this. Prettier doesn't seem to follow the style, and just keeps everything as is

@MonicaOlejniczak MonicaOlejniczak merged commit 524ed7e into v2 Feb 29, 2024
14 of 16 checks passed
@mischnic mischnic deleted the molejniczak/ci-cache branch March 1, 2024 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants