- 
                Notifications
    
You must be signed in to change notification settings  - Fork 24
 
feat: Add just task runner to ensure reproducibility of CI #1345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
          
     Open
      
      
            maru-ava
  wants to merge
  12
  commits into
  main
  
    
      
        
          
  
    
      Choose a base branch
      
     
    
      
        
      
      
        
          
          
        
        
          
            
              
              
              
  
           
        
        
          
            
              
              
           
        
       
     
  
        
          
            
          
            
          
        
       
    
      
from
maru/just-ffi
  
      
      
   
  
    
  
  
  
 
  
      
    base: main
Could not load branches
            
              
  
    Branch not found: {{ refName }}
  
            
                
      Loading
              
            Could not load tags
            
            
              Nothing to show
            
              
  
            
                
      Loading
              
            Are you sure you want to change the base?
            Some commits from the old base branch may be removed from the timeline,
            and old review comments may become outdated.
          
          
      
        
          +177
        
        
          −18
        
        
          
        
      
    
  
  
     Open
                    Changes from all commits
      Commits
    
    
            Show all changes
          
          
            12 commits
          
        
        Select commit
          Hold shift + click to select a range
      
      2e6a32a
              
                Ensure golang input is up-to-date for nix flake
              
              
                maru-ava 7951296
              
                fixup: Update to use the non-deprecated flake update syntax
              
              
                maru-ava 0dc23b7
              
                feat: Add just task runner to ensure reproducibility of CI
              
              
                maru-ava f544432
              
                fixup: Exclude justfile and run-just.sh for licence header check
              
              
                maru-ava 0ce83e7
              
                fixup: Add output to differentiate test recipes
              
              
                maru-ava e3493bc
              
                fixup: Use flake path as intended
              
              
                maru-ava 262b823
              
                fixup: Update to use current flake update syntax
              
              
                maru-ava efc90b6
              
                fixup: Add separate command for updating the nix flake
              
              
                maru-ava 734c3c7
              
                fixup: Move run script and fix update flake command name
              
              
                maru-ava 86ab4ed
              
                fixup: Add check for golang version consistency
              
              
                maru-ava b93f0da
              
                fixup: Cleanup justfile comments
              
              
                maru-ava 846c5cd
              
                Merge branch 'main' into maru/just-ffi
              
              
                rkuris File filter
Filter by extension
Conversations
          Failed to load comments.   
        
        
          
      Loading
        
  Jump to
        
          Jump to file
        
      
      
          Failed to load files.   
        
        
          
      Loading
        
  Diff view
Diff view
There are no files selected for viewing
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              | Original file line number | Diff line number | Diff line change | 
|---|---|---|
| 
          
            
          
           | 
    @@ -53,6 +53,8 @@ | |
| "cliff.toml", | ||
| "clippy.toml", | ||
| "**/tests/compile_*/**", | ||
| "justfile", | ||
| "scripts/run-just.sh", | ||
| ], | ||
| } | ||
| ] | ||
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
      
      Oops, something went wrong.
      
    
  
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              
              
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              | Original file line number | Diff line number | Diff line change | 
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| # List available recipes | ||
| default: | ||
| ./scripts/run-just.sh --list | ||
| 
     | 
||
| # Build ffi with nix | ||
| build-ffi-nix: check-nix | ||
| cd ffi && nix build | ||
| 
     | 
||
| # Check if the git branch is clean | ||
| check-clean-branch: | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| 
     | 
||
| git add --all | ||
| git update-index --really-refresh >> /dev/null | ||
| 
     | 
||
| # Show the status of the working tree. | ||
| git status --short | ||
| 
     | 
||
| # Exits if any uncommitted changes are found. | ||
| git diff-index --quiet HEAD | ||
| 
     | 
||
| # Check if the FFI flake is up-to-date (requires clean git tree) | ||
| check-ffi-flake: check-nix | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| ./scripts/run-just.sh update-ffi-flake | ||
| ./scripts/run-just.sh check-clean-branch | ||
| 
     | 
||
| # Check if the golang version is set consistently (requires clean git tree) | ||
| check-golang-version: check-nix | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| 
     | 
||
| # Exit only at the end if any of the checks set FAILED=1 | ||
| FAILED= | ||
| 
     | 
||
| cd ffi | ||
| 
     | 
||
| TOOLCHAIN_VERSION=$(nix develop --command bash -c "go mod edit -json | jq -r '.Toolchain'") | ||
| echo "toolchain version in ffi/go.mod is ${TOOLCHAIN_VERSION}" | ||
| 
     | 
||
| ETH_TESTS_VERSION=$(nix develop --command bash -c "cd tests/eth && go mod edit -json | jq -r '.Toolchain'") | ||
| echo "toolchain version in ffi/tests/eth/go.mod is ${ETH_TESTS_VERSION}" | ||
| 
     | 
||
| if [[ "${TOOLCHAIN_VERSION}" != "${ETH_TESTS_VERSION}" ]]; then | ||
| echo "❌ toolchain version in ffi/tests/eth/go.mod should be ${TOOLCHAIN_VERSION}" | ||
| FAILED=1 | ||
| fi | ||
| 
     | 
||
| FIREWOOD_TESTS_VERSION=$(nix develop --command bash -c "cd tests/firewood && go mod edit -json | jq -r '.Toolchain'") | ||
| echo "toolchain version in ffi/tests/firewood/go.mod is ${FIREWOOD_TESTS_VERSION}" | ||
| 
     | 
||
| if [[ "${TOOLCHAIN_VERSION}" != "${FIREWOOD_TESTS_VERSION}" ]]; then | ||
| echo "❌ toolchain version in ffi/tests/firewood/go.mod should be ${TOOLCHAIN_VERSION}" | ||
| FAILED=1 | ||
| fi | ||
| 
     | 
||
| NIX_VERSION=$(nix run .#go -- version | awk '{print $3}') | ||
| echo "golang provided by ffi/flake.nix is ${NIX_VERSION}" | ||
| 
     | 
||
| if [[ "${TOOLCHAIN_VERSION}" != "${NIX_VERSION}" ]]; then | ||
| echo "❌ golang provided by ffi/flake/nix should be ${TOOLCHAIN_VERSION}" | ||
| echo "It will be necessary to update the golang.url in ffi/flake.nix to point to a SHA of"\ | ||
| "AvalancheGo whose nix/go/flake.nix provides ${TOOLCHAIN_VERSION}." | ||
| fi | ||
| 
     | 
||
| if [[ -n "${FAILED}" ]]; then | ||
| exit 1 | ||
| fi | ||
| 
     | 
||
| # Check if nix is installed | ||
| check-nix: | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| if ! command -v nix &> /dev/null; then | ||
| echo "Error: 'nix' is not installed." >&2 | ||
| echo "" >&2 | ||
| echo "To install nix:" >&2 | ||
| echo " - Visit: https://github.com/DeterminateSystems/nix-installer" >&2 | ||
| echo " - Or run: curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install" >&2 | ||
| exit 1 | ||
| fi | ||
| 
     | 
||
| # Run all checks of ffi built with nix | ||
| test-ffi-nix: test-ffi-nix-build-equivalency test-ffi-nix-go-bindings | ||
| 
     | 
||
| # Test ffi build equivalency between nix and cargo | ||
| test-ffi-nix-build-equivalency: check-nix | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| 
     | 
||
| echo "Testing ffi build equivalency between nix and cargo" | ||
| 
     | 
||
| bash -x ./ffi/test-build-equivalency.sh | ||
| 
     | 
||
| # Test golang ffi bindings using the nix-built artifacts | ||
| test-ffi-nix-go-bindings: build-ffi-nix | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| 
     | 
||
| echo "running ffi tests against bindings built by nix..." | ||
| 
     | 
||
| cd ffi | ||
| 
     | 
||
| # Need to capture the flake path before changing directories to | ||
| # result/ffi because `result` is a nix store symlink so ../../ | ||
| # won't resolve to the ffi path containing the flake. | ||
| FLAKE_PATH="$PWD" | ||
| 
     | 
||
| # This runs golang outside a nix shell to validate viability | ||
| # without the env setup performed by a nix shell | ||
| GO="nix run $FLAKE_PATH#go" | ||
| 
     | 
||
| cd result/ffi | ||
| 
     | 
||
| # - cgocheck2 is expensive but provides complete pointer checks | ||
| # - use hash mode ethhash since the flake builds with `--features ethhash,logger` | ||
| GOEXPERIMENT=cgocheck2 TEST_FIREWOOD_HASH_MODE=ethhash ${GO} test ./... | ||
| 
     | 
||
| # Ensure the FFI flake is up-to-date | ||
| update-ffi-flake: check-nix | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| cd ffi | ||
| 
     | 
||
| echo "ensuring flake lock file is current for golang" | ||
| nix flake update golang | ||
| 
     | 
||
| echo "checking for a consistent golang verion" | ||
| ../scripts/run-just.sh check-golang-version | 
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              | Original file line number | Diff line number | Diff line change | 
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| 
     | 
||
| if command -v just &> /dev/null; then | ||
| exec just "$@" | ||
| elif command -v nix &> /dev/null; then | ||
| exec nix run nixpkgs#just -- "$@" | ||
| else | ||
| echo "Error: Neither 'just' nor 'nix' is installed." >&2 | ||
| echo "" >&2 | ||
| echo "Please install one of the following:" >&2 | ||
| echo "" >&2 | ||
| echo "Option 1 - Install just:" >&2 | ||
| echo " - Visit: https://github.com/casey/just#installation" >&2 | ||
| echo " - Or use cargo: cargo install just" >&2 | ||
| echo "" >&2 | ||
| echo "Option 2 - Install nix:" >&2 | ||
| echo " - Visit: https://github.com/DeterminateSystems/nix-installer" >&2 | ||
| echo " - Or run: curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install" >&2 | ||
| exit 1 | ||
| fi | 
      
      Oops, something went wrong.
        
    
  
  Add this suggestion to a batch that can be applied as a single commit.
  This suggestion is invalid because no changes were made to the code.
  Suggestions cannot be applied while the pull request is closed.
  Suggestions cannot be applied while viewing a subset of changes.
  Only one suggestion per line can be applied in a batch.
  Add this suggestion to a batch that can be applied as a single commit.
  Applying suggestions on deleted lines is not supported.
  You must change the existing code in this line in order to create a valid suggestion.
  Outdated suggestions cannot be applied.
  This suggestion has been applied or marked resolved.
  Suggestions cannot be applied from pending reviews.
  Suggestions cannot be applied on multi-line comments.
  Suggestions cannot be applied while the pull request is queued to merge.
  Suggestion cannot be applied right now. Please check back later.
  
    
  
    
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used by the
check-golang-versionrecipe.