Skip to content

pytest.md

Latest commit

 

History

History
77 lines (56 loc) · 3.61 KB

pytest.md

File metadata and controls

77 lines (56 loc) · 3.61 KB

Using bktec with pytest

To integrate bktec with pytest, you need to install and configure Buildkite Test Collector for pytest first. Then set the BUILDKITE_TEST_ENGINE_TEST_RUNNER environment variable to pytest.

export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
bktec run

Configure test command

By default, bktec runs pytest with the following command:

pytest {{testExamples}} --json={{resultPath}}

In this command, {{testExamples}} is replaced by bktec with the list of test files or tests to run, and {{resultPath}} is replaced with a unique temporary path created by bktec. --json option is a custom option added by Buildkite Test Collector to save the result into a JSON file at given path. You can customize this command using the BUILDKITE_TEST_ENGINE_TEST_CMD environment variable.

To customize the test command, set the following environment variable:

export BUILDKITE_TEST_ENGINE_TEST_CMD="pytest --cache-clear --json={{resultPath}} {{testExamples}}"

Important

Make sure to include --json={{resultPath}} in your custom test command, as bktec requires this to read the test results for retries and verification purposes.

Filter test files

By default, bktec runs test files that match the **/{*_test,test_*}.py pattern. You can customize this pattern using the BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN environment variable. For instance, to configure bktec to only run test files inside the tests directory, use:

export BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN="tests/**/{*_test,test_*}.py"

Additionally, you can exclude specific files or directories that match a certain pattern using the BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN environment variable. For example, to exclude test files inside the tests/api directory, use:

export BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN=tests/api

You can also use both BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN and BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN simultaneously. For example, to run all test files inside the tests/ directory, except those inside tests/api, use:

export BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN="**/{*_test,test_*}.py"
export BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN=tests/api

Tip

This option accepts the pattern syntax supported by the zzglob library.

Filter test by tags

You can filter tests to run based on execution_tag markers using the BUILDKITE_TEST_ENGINE_TAG_FILTERS environment variable or --tag-filters CLI option.

import pytest

@pytest.mark.execution_tag('gpu_allocation', '2')
def test_my_test():
    ...

To run tests tagged with gpu_allocation of 2:

export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
export BUILDKITE_TEST_ENGINE_TAG_FILTERS="gpu_allocation:2"
bktec run

Note

execution_tag is a custom marker added by Buildkite Test Collector for pytest. You can add multiple tags to a test, however the tag filters only support single key:value pairs.

Automatically retry failed tests

You can configure bktec to automatically retry failed tests using the BUILDKITE_TEST_ENGINE_RETRY_COUNT environment variable. When this variable is set to a number greater than 0, bktec will retry each failed test up to the specified number of times, using either the default test command or the command specified in BUILDKITE_TEST_ENGINE_TEST_CMD.

To enable automatic retry, set the following environment variable:

export BUILDKITE_TEST_ENGINE_RETRY_COUNT=2