Skip to content

Commit a5026e9

Browse files
aaronbuchwaldaaronbuchwald
authored
Add GitHub Action workflow to generate and attach static libraries to source code for golang FFI (#868)
1 parent 59a2299 commit a5026e9

File tree

5 files changed

+231
-10
lines changed

5 files changed

+231
-10
lines changed

‎.github/workflows/attach-static-libs.yaml‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
name: attach-static-libs
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
create_branch_name:
7+
description: "Name of the new branch to create and attach static libs"
8+
required: true
9+
push:
10+
tags:
11+
- "*"
12+
pull_request:
13+
14+
env:
15+
CARGO_TERM_COLOR: always
16+
17+
# Build, upload, and collect static libraries for each target architecture,
18+
# so that golang projects can import the FFI package without needing to
19+
# recompile Firewood locally.
20+
# Supported architectures are:
21+
# - x86_64-unknown-linux-gnu
22+
# - aarch64-unknown-linux-gnu
23+
# - x86_64-apple-darwin
24+
# - aarch64-apple-darwin
25+
jobs:
26+
# Build the static libraries for each target architecture and upload
27+
# them as artifacts to collect and attach in the next job.
28+
build-firewood-ffi-libs:
29+
runs-on: ${{ matrix.os }}
30+
strategy:
31+
matrix:
32+
include:
33+
- os: ubuntu-latest
34+
target: x86_64-unknown-linux-gnu
35+
- os: ubuntu-22.04-arm
36+
target: aarch64-unknown-linux-gnu
37+
- os: macos-latest
38+
target: aarch64-apple-darwin
39+
- os: macos-13
40+
target: x86_64-apple-darwin
41+
steps:
42+
- uses: actions/checkout@v4
43+
- uses: dtolnay/rust-toolchain@stable
44+
- uses: arduino/setup-protoc@v3
45+
with:
46+
repo-token: ${{ secrets.GITHUB_TOKEN }}
47+
- uses: Swatinem/rust-cache@v2
48+
49+
- name: Build for ${{ matrix.target }}
50+
# TODO: add ethhash feature flag after updating FFI tests
51+
run: cargo build --profile maxperf --features logger --target ${{ matrix.target }} -p firewood-ffi
52+
53+
- name: Upload binary
54+
uses: actions/upload-artifact@v4
55+
with:
56+
name: ${{ matrix.target }}
57+
path: target/${{ matrix.target }}/maxperf/libfirewood_ffi.a
58+
if-no-files-found: error
59+
60+
# Collect all the static libraries built on the previous matrix of jobs
61+
# and add them into ffi/libs directory.
62+
# We commit and push this as a new branch with "--force" to overwrite
63+
# the previous static libs that will not be on our branch.
64+
push-firewood-ffi-libs:
65+
needs: build-firewood-ffi-libs
66+
runs-on: ubuntu-latest
67+
outputs:
68+
target_branch: ${{ steps.determine_branch.outputs.target_branch }}
69+
steps:
70+
- name: Determine branch name
71+
id: determine_branch
72+
run: |
73+
if [[ "${{ github.event_name }}" == "workflow_dispatch" && -n "${{ github.event.inputs.create_branch_name }}" ]]; then
74+
export target_branch="${{ github.event.inputs.create_branch_name }}"
75+
echo "Using workflow input as target branch: $target_branch"
76+
echo "target_branch=$target_branch" >> "$GITHUB_OUTPUT"
77+
elif [[ "${{ github.event_name }}" == "push" ]]; then
78+
export target_branch="${GITHUB_REF#refs/heads/}"
79+
echo "Using tag/branch name as target_branch: $target_branch"
80+
echo "target_branch=$target_branch" >> "$GITHUB_OUTPUT"
81+
elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
82+
export target_branch="${{ github.event.pull_request.head.ref }}"
83+
echo "Using PR head name as target branch: $target_branch"
84+
echo "target_branch=$target_branch" >> "$GITHUB_OUTPUT"
85+
else
86+
echo "No valid input or tag found."
87+
exit 1
88+
fi
89+
90+
- uses: actions/checkout@v4
91+
with:
92+
path: firewood
93+
94+
- uses: actions/checkout@v4
95+
with:
96+
repository: ava-labs/firewood-go
97+
token: ${{ secrets.FIREWOOD_GO_GITHUB_TOKEN }}
98+
path: firewood-go
99+
100+
- name: Copy FFI Source Code
101+
run: cp -r firewood/ffi firewood-go
102+
103+
- name: Download binaries into libs directory
104+
uses: actions/download-artifact@v4
105+
with:
106+
path: firewood-go/ffi/libs
107+
108+
- name: List downloaded target directory
109+
run: find firewood-go -type f | sort
110+
111+
- name: Push static libs to branch
112+
working-directory: firewood-go
113+
# GITHUB_TOKEN is configured in the last actions/checkout step
114+
# to have read/write permissions to the firewood-go repo.
115+
run: |
116+
git config --global user.name "FirewoodCI"
117+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
118+
git checkout -b ${{ steps.determine_branch.outputs.target_branch }}
119+
git add .
120+
git commit -m "firewood ci ${{ github.sha }}: attach firewood static libs"
121+
git push -u origin ${{ steps.determine_branch.outputs.target_branch }} --force
122+
if [[ "${{ github.ref_type }}" == "tag" ]]; then
123+
git tag -a "${GITHUB_REF#refs/tags/}" -m "firewood ci ${{ github.sha }}: attach firewood static libs"
124+
git push origin "${GITHUB_REF#refs/tags/}"
125+
fi
126+
127+
# Check out the branch created in the previous job on a matrix of
128+
# our target architectures and test the FFI package on a fresh
129+
# machine without re-compiling Firewood locally.
130+
# This tests that the Firewood FFI package passes tests on the target
131+
# architecture when it is forced to depend on the attached static libs.
132+
test-firewood-ffi-libs:
133+
runs-on: ${{ matrix.os }}
134+
strategy:
135+
matrix:
136+
os: [ubuntu-latest, ubuntu-22.04-arm, macos-latest, macos-13]
137+
needs: push-firewood-ffi-libs
138+
continue-on-error: true
139+
steps:
140+
- uses: actions/checkout@v4
141+
with:
142+
repository: ava-labs/firewood-go
143+
token: ${{ secrets.FIREWOOD_GO_GITHUB_TOKEN }}
144+
ref: ${{ needs.push-firewood-ffi-libs.outputs.target_branch }}
145+
- name: Set up Go
146+
uses: actions/setup-go@v5
147+
with:
148+
go-version-file: "ffi/go.mod"
149+
cache-dependency-path: "ffi/go.sum"
150+
- name: Test Go FFI bindings
151+
working-directory: ffi
152+
# cgocheck2 is expensive but provides complete pointer checks
153+
run: GOEXPERIMENT=cgocheck2 go test ./...
154+
155+
remove-if-pr-only:
156+
runs-on: ubuntu-latest
157+
needs: [push-firewood-ffi-libs, test-firewood-ffi-libs]
158+
if: needs.push-firewood-ffi-libs.result == 'success' && github.event_name == 'pull_request'
159+
permissions:
160+
# Give the GITHUB_TOKEN write permission to delete the
161+
# branch created by the previous job if it is a pull request.
162+
contents: write
163+
steps:
164+
- uses: actions/checkout@v4
165+
with:
166+
repository: ava-labs/firewood-go
167+
token: ${{ secrets.FIREWOOD_GO_GITHUB_TOKEN }}
168+
ref: ${{ needs.push-firewood-ffi-libs.outputs.target_branch }}
169+
- name: Delete branch
170+
run: |
171+
git push origin --delete ${{ needs.push-firewood-ffi-libs.outputs.target_branch }}

‎.github/workflows/ci.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@ name: ci
22

33
on:
44
pull_request:
5-
branches: ['*']
65
push:
76
branches: [main]
87

@@ -106,7 +105,10 @@ jobs:
106105

107106
ffi:
108107
needs: build
109-
runs-on: ubuntu-latest
108+
runs-on: ${{ matrix.os }}
109+
strategy:
110+
matrix:
111+
os: [ubuntu-latest, macos-latest]
110112
steps:
111113
- uses: actions/checkout@v4
112114
- uses: dtolnay/rust-toolchain@stable

‎ffi/README.md‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,52 @@
1-
# A firewood golang interface
1+
# Firewood Golang FFI
22

3-
This allows calling into firewood from golang
3+
The FFI package provides a golang FFI layer for Firewood.
44

5-
## Building
5+
## Building Firewood Golang FFI
66

7-
First, build the release version (`cargo build --release`). This creates the ffi
8-
interface file "firewood.h" as a side effect.
7+
The Golang FFI layer uses a CGO directive to locate a C-API compatible binary built from Firewood. Firewood supports both seamless local development and a single-step compilation process for Go projects that depend or transitively depend on Firewood.
98

10-
Then, you can run the tests in go, using `go test .`
9+
To do this, [firewood.go](./firewood.go) includes CGO directives to include multiple search paths for the Firewood binary in the local `target/` build directory and `ffi/libs`. For the latter, [attach-static-libs](../.github/workflows/attach-static-libs.yaml) GitHub Action pushes an FFI package with static libraries attached for the following supported architectures:
10+
11+
- x86_64-unknown-linux-gnu
12+
- aarch64-unknown-linux-gnu
13+
- aarch64-apple-darwin
14+
- x86_64-apple-darwin
15+
16+
to a separate repo [firewood-go](https://github.com/ava-labs/firewood-go) (to avoid including binaries in the Firewood repo).
17+
18+
### Local Development
19+
20+
[firewood.go](./firewood.go) includes CGO directives to include builds in the `target/` directory.
21+
22+
Firewood prioritizes builds in the following order:
23+
24+
1. maxperf
25+
2. release
26+
3. debug
27+
28+
To use and test the Firewood FFI locally, you can run:
29+
30+
```bash
31+
cargo build --profile maxperf
32+
cd ffi
33+
go test
34+
```
35+
36+
To use a local build of Firewood for a project that depends on Firewood, you must redirect the `go.mod` to use the local version of Firewood FFI, for example:
37+
38+
```bash
39+
go mod edit -replace github.com/ava-labs/firewood-go/ffi=/path/to/firewood/ffi
40+
go mod tidy
41+
```
42+
43+
### Production Development Flow
44+
45+
Firewood pushes the FFI source code and attached static libraries to [firewood-go](https://github.com/ava-labs/firewood-go) via [attach-static-libs](../.github/workflows/attach-static-libs.yaml).
46+
47+
This enables consumers to utilize it directly without forcing them to compile Firewood locally. Go programs running on supported architectures can utilize `firewood-go/ffi` just like any other dependency.
48+
49+
To trigger this build, [attach-static-libs](../.github/workflows/attach-static-libs.yaml) supports triggers for both manual GitHub Actions and tags, so you can create a mirror branch/tag on [firewood-go](https://github.com/ava-labs/firewood-go) by either trigger a manual GitHub Action and selecting your branch or pushing a tag to Firewood.
1150

1251
## Development
1352
Iterative building is unintuitive for the ffi and some common sources of confusion are listed below.

‎ffi/firewood.go‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,16 @@
44
package firewood
55

66
// // Note that -lm is required on Linux but not on Mac.
7-
// #cgo LDFLAGS: -L${SRCDIR}/../target/release -L/usr/local/lib -lfirewood_ffi -lm
7+
// #cgo linux,amd64 LDFLAGS: -L${SRCDIR}/libs/x86_64-unknown-linux-gnu -lm
8+
// #cgo linux,arm64 LDFLAGS: -L${SRCDIR}/libs/aarch64-unknown-linux-gnu -lm
9+
// #cgo darwin,amd64 LDFLAGS: -L${SRCDIR}/libs/x86_64-apple-darwin
10+
// #cgo darwin,arm64 LDFLAGS: -L${SRCDIR}/libs/aarch64-apple-darwin
11+
// // XXX: last search path takes precedence, which means we prioritize
12+
// // local builds over pre-built and maxperf over release build
13+
// #cgo LDFLAGS: -L${SRCDIR}/../target/debug
14+
// #cgo LDFLAGS: -L${SRCDIR}/../target/release
15+
// #cgo LDFLAGS: -L${SRCDIR}/../target/maxperf
16+
// #cgo LDFLAGS: -L/usr/local/lib -lfirewood_ffi
817
// #include <stdlib.h>
918
// #include "firewood.h"
1019
import "C"

‎ffi/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -583,7 +583,7 @@ impl Display for Value {
583583
match (self.len, self.data.is_null()) {
584584
(0, true) => write!(f, "[not found]"),
585585
(0, false) => write!(f, "[error] {}", unsafe {
586-
CStr::from_ptr(self.data.cast::<i8>()).to_string_lossy()
586+
CStr::from_ptr(self.data.cast::<c_char>()).to_string_lossy()
587587
}),
588588
(len, true) => write!(f, "[id] {len}"),
589589
(_, false) => write!(f, "[data] {:?}", self.as_slice()),

0 commit comments

Comments
 (0)