BigQuery Buildkite Plugin

This Buildkite plugin can be used to deploy tables/views schemas to BigQuery

A Buildkite plugin for deploying tables/views schemas to BigQuery.

Using the plugin

If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as master or any branch names.

Simple

steps:
  - plugins:
      - wayfair-incubator/bigquery#v1.2.2:
          gcp_project: gcp-us-project
          dataset_schema_directory: schemas/gcp-us-project/dataset

Configuration

Required

gcp_project (required, string)

The full name of the GCP project you want to deploy.

Example: gcp-us-project

dataset_schema_directory (required, string)

The directory in your repository where are you storing the schemas for your tables and views.

Example: gcp-us-project/dataset_name

prod_build_branch (optional, string)

Default branch for the repo.

Example: main

Default: master

execute_only_changed_files (optional, boolean)

Whether to execute just the files that changed in a directory or not.

Example: true

Default: true

fail_pipeline_on_first_exception (optional, boolean)

Whether to fail pipeline as soon as there is one failure.

Example: true

Default: true

Secret

This plugin expects GCP_SERVICE_ACCOUNT is placed as environment variable. Make sure to store it securely!

env:
  gcp_service_account: '{"email": ""}'

Example

Basic

The following pipeline will deploy all the schemas to the gcp-us-project living under directory/project/

Schemas

This plugin uses GBQ to deploy to Google BigQuery. GBQ now supports specifying partitions with the schema as well.

To leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.

All the configuration options can be found here.

{
  "partition": {
    "type": "range",
    "definition": {
      "field": "ID",
      "range": {
        "start": 1,
        "end": 100000,
        "interval": 10
      }
    }
  },
  "clustering": [
    "ID"
  ],
  "schema": [
    {
      "name": "ID",
      "type": "INTEGER",
      "mode": "REQUIRED"
    }
  ]
}

Contributing

See the Contributing Guide for additional information.

To execute tests locally (requires that docker and docker-compose are installed):

docker-compose run test

Development Commands

The project uses uv for dependency management and ruff for linting/formatting.

# Run all tests and linting with auto-formatting
docker-compose run --rm py-test

# Run tests only
docker-compose run --rm devbox pytest

# Run ruff format (auto-fix formatting issues)
docker-compose run --rm ruff-format

# Run ruff linting with auto-fix
docker-compose run --rm ruff-lint

# Run ruff check only (no auto-fix)
docker-compose run --rm ruff-check

# Run mypy type checking
docker-compose run --rm mypy

# Lock dependencies after updating requirements.txt
docker-compose run --rm lock-requirements
docker-compose build devbox

Credits

This plugin was originally written by Jash Parekh for Wayfair.