Skip to content

Commit 52c9d48

Browse files
nicktrnnicktrn
authored
v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url * add maybe platform down error * shorten config path * extend protobuf compiler install instructions * build command and image model * new image triggers task indexing * index support for prod builds * shorten example provider secret * multi-stage task prod build * lock prod tasks to node 18 image * extract shared build and dev command libs * pull out more shared deps * add coordinator and providers * fix core-apps build * add dev builds for new apps * fix cwd * enable corepack * some build fixes * enable buildkit for old docker versions * coordinator image fixes * update provider containerfile * build dev images in parallel * upgrade pgadmin * fix prod facade build * prod runs * fix merge * don't knock out platform on invalid attempt id * fix prod facade * rename to build.ts * fix prod builds * set to executing after fetching payload * make prod worker listen on random port if in use * remove experimental warnings in dev * prod resume * prevent execution after completion * exit prod worker after completion * always restart otel collector * docker checkpoints and prod runtime messaging * don't retry indexing without chance of success * make platform checkpoint aware * deploy with existing hash sets latest worker * log restore requests * only try to checkpoint long waits * tidying up * lockfile * fix build * prod worker merge fixes * fix prod complete and cancel * fix lua nil checks * socket namespace abstraction * cleanup * make all build args optional * add build script * don't require env vars for dev * fix schema * prod merge * small fix * bind correct logger * fix v3 ref catalog entry * resume prod batch * pass socket to error and disconnect handlers * fix non-batch resume * fix batch resume * send connection env vars when not in dev * create worker via socket * move api client back into v3 cli * fix lockfile * fix resume with failures * marqs replace message * typecheck prior to build * don't define api url in prod builds * support prod retries after resume * skip typecheck option
1 parent ca47e6b commit 52c9d48

File tree

95 files changed

+7255
-448
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

95 files changed

+7255
-448
lines changed

‎.env.example

+5-1
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,8 @@ CLOUD_LINEAR_CLIENT_ID=
4545
CLOUD_LINEAR_CLIENT_SECRET=
4646
CLOUD_SLACK_APP_HOST=
4747
CLOUD_SLACK_CLIENT_ID=
48-
CLOUD_SLACK_CLIENT_SECRET=
48+
CLOUD_SLACK_CLIENT_SECRET=
49+
50+
# v3 variables
51+
PROVIDER_SECRET=provider-secret # generate the actual secret with `openssl rand -hex 32`
52+
COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl rand -hex 32`

‎.github/workflows/publish-dev.yml

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: "🚢 Publish Container Images (dev)"
2+
3+
on:
4+
push:
5+
tags:
6+
- "dev-*"
7+
paths:
8+
- ".github/workflows/publish.yml"
9+
- "packages/**"
10+
- "!packages/**/*.md"
11+
- "!packages/**/*.eslintrc"
12+
- "apps/**"
13+
- "!apps/**/*.md"
14+
- "!apps/**/*.eslintrc"
15+
- "integrations/**"
16+
- "!integrations/**/*.md"
17+
- "!integrations/**/*.eslintrc"
18+
- "pnpm-lock.yaml"
19+
- "pnpm-workspace.yaml"
20+
- "turbo.json"
21+
- "docker/Dockerfile"
22+
- "docker/scripts/**"
23+
- "tests/**"
24+
25+
permissions:
26+
id-token: write
27+
packages: write
28+
contents: read
29+
30+
concurrency:
31+
group: ${{ github.workflow }}-${{ github.ref }}
32+
33+
env:
34+
AWS_REGION: us-east-1
35+
36+
jobs:
37+
build:
38+
strategy:
39+
matrix:
40+
package: [coordinator, kubernetes-provider]
41+
runs-on: buildjet-4vcpu-ubuntu-2204
42+
env:
43+
DOCKER_BUILDKIT: "1"
44+
steps:
45+
- uses: actions/checkout@v4
46+
47+
- name: Generate build ID
48+
id: prep
49+
run: |
50+
sha=${GITHUB_SHA::7}
51+
ts=$(date +%s)
52+
echo "BUILD_ID=${{ matrix.package }}-${sha}-${ts}" >> "$GITHUB_OUTPUT"
53+
54+
- name: Set up Docker Buildx
55+
uses: docker/setup-buildx-action@v2
56+
57+
# ..to avoid rate limits when pulling images
58+
- name: Login to DockerHub
59+
uses: docker/login-action@v2
60+
with:
61+
username: ${{ secrets.DOCKERHUB_USERNAME }}
62+
password: ${{ secrets.DOCKERHUB_TOKEN }}
63+
64+
- name: 🚢 Build Container Image
65+
run: |
66+
docker build -t dev_image -f ./apps/${{ matrix.package }}/Containerfile .
67+
68+
# ..to push image
69+
- name: 🐙 Login to GitHub Container Registry
70+
uses: docker/login-action@v2
71+
with:
72+
registry: ghcr.io
73+
username: ${{ github.repository_owner }}
74+
password: ${{ secrets.GITHUB_TOKEN }}
75+
76+
- name: 🐙 Push to GitHub Container Registry
77+
run: |
78+
docker tag dev_image $REGISTRY/$REPOSITORY:$IMAGE_TAG
79+
docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
80+
env:
81+
REGISTRY: ghcr.io/triggerdotdev
82+
REPOSITORY: dev
83+
IMAGE_TAG: ${{ steps.prep.outputs.BUILD_ID }}

‎apps/coordinator/.env.example

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
HTTP_SERVER_PORT=8020
2+
3+
PLATFORM_ENABLED=true
4+
PLATFORM_WS_PORT=3030

‎apps/coordinator/.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dist/
2+
node_modules/
3+
.env

‎apps/coordinator/Containerfile

+63
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# syntax=docker/dockerfile:labs
2+
3+
FROM node:18.18.2-bullseye-slim@sha256:21479df46c3173ee0cefc6b264928e10239152c4f74df872ca9369be01a245b7 AS node-18
4+
5+
WORKDIR /app
6+
7+
FROM node-18 AS pruner
8+
9+
COPY --chown=node:node . .
10+
RUN npx -q [email protected] prune --scope=coordinator --docker
11+
RUN find . -name "node_modules" -type d -prune -exec rm -rf '{}' +
12+
13+
FROM node-18 AS base
14+
15+
RUN apt-get update \
16+
&& apt-get install -y buildah ca-certificates dumb-init \
17+
&& rm -rf /var/lib/apt/lists/*
18+
19+
COPY --chown=node:node .gitignore .gitignore
20+
COPY --from=pruner --chown=node:node /app/out/json/ .
21+
COPY --from=pruner --chown=node:node /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
22+
COPY --from=pruner --chown=node:node /app/out/pnpm-workspace.yaml ./pnpm-workspace.yaml
23+
24+
FROM base AS dev-deps
25+
RUN corepack enable
26+
ENV NODE_ENV development
27+
28+
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm fetch --no-frozen-lockfile
29+
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --ignore-scripts --no-frozen-lockfile
30+
31+
FROM base AS builder
32+
RUN corepack enable
33+
34+
COPY --from=pruner --chown=node:node /app/out/full/ .
35+
COPY --from=dev-deps --chown=node:node /app/ .
36+
COPY --chown=node:node turbo.json turbo.json
37+
38+
RUN pnpm run -r --filter '@trigger.dev/core*' build
39+
RUN pnpm run -r --filter coordinator build:bundle
40+
41+
FROM alpine AS cri-tools
42+
43+
WORKDIR /cri-tools
44+
45+
ARG CRICTL_VERSION=v1.29.0
46+
ARG CRICTL_CHECKSUM=sha256:d16a1ffb3938f5a19d5c8f45d363bd091ef89c0bc4d44ad16b933eede32fdcbb
47+
ADD --checksum=${CRICTL_CHECKSUM} \
48+
https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz .
49+
RUN tar zxvf crictl-${CRICTL_VERSION}-linux-amd64.tar.gz
50+
51+
FROM base AS runner
52+
53+
RUN corepack enable
54+
ENV NODE_ENV production
55+
56+
COPY --from=cri-tools --chown=node:node /cri-tools/crictl /usr/local/bin
57+
COPY --from=builder --chown=node:node /app/apps/coordinator/dist/index.cjs ./index.cjs
58+
59+
EXPOSE 8000
60+
61+
USER node
62+
63+
CMD [ "/usr/bin/dumb-init", "--", "/usr/local/bin/node", "./index.cjs" ]

‎apps/coordinator/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Coordinator
2+
3+
Sits between the platform and tasks. Facilitates communication and checkpointing, amongst other things.

‎apps/coordinator/package.json

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"name": "coordinator",
3+
"private": true,
4+
"version": "0.0.1",
5+
"description": "",
6+
"main": "dist/index.cjs",
7+
"type": "module",
8+
"scripts": {
9+
"build": "npm run build:bundle",
10+
"build:bundle": "esbuild src/index.ts --bundle --outfile=dist/index.cjs --platform=node",
11+
"build:image": "docker build -f Containerfile . -t coordinator",
12+
"dev": "tsx --no-warnings=ExperimentalWarning --require dotenv/config --watch src/index.ts",
13+
"start": "tsx src/index.ts",
14+
"typecheck": "tsc --noEmit"
15+
},
16+
"keywords": [],
17+
"author": "",
18+
"license": "MIT",
19+
"dependencies": {
20+
"@trigger.dev/core": "workspace:*",
21+
"@trigger.dev/core-apps": "workspace:*",
22+
"execa": "^8.0.1",
23+
"prom-client": "^15.1.0",
24+
"socket.io": "^4.7.4",
25+
"socket.io-client": "^4.7.4"
26+
},
27+
"devDependencies": {
28+
"@types/node": "^18",
29+
"dotenv": "^16.4.2",
30+
"esbuild": "^0.19.11",
31+
"tsx": "^4.7.0",
32+
"typescript": "^5.3.3"
33+
}
34+
}

0 commit comments

Comments
 (0)