Skip to content

Conversation

@nprizal
Copy link
Contributor

@nprizal nprizal commented Jan 22, 2026

Test Engine Tags can be applied to a test using @pytest.mark.execution_tag(“key”, “value”). However, pytest built-in -m option doesn’t support filtering tests using marker arguments. It can filter tests that have any execution_tag markers, but it can't filter tests that have execution_tag marker with specific key and value.

This PR adds new custom options --tag-filters to filter tests based on execution_tag'skey & value.

# test_foobar.py
import pytest

@pytest.mark.execution_tag("color", "red")
def test_foo()
 assert True

@pytest.mark.execution_tag("color", "blue")
def test_bar()
  assert True

Then to filter tests with execution tag

$ pytest --tag-filters "color:blue" -v
=========================== test session starts ============================
platform darwin -- Python 3.13.2, pytest-9.0.2, pluggy-1.6.0
rootdir: /my/project
plugins: buildkite-test-collector-1.2.0
collected 2 items / 1 deselected / 1 selected

test_foobar.py::test_bar PASSED                                       [100%]

===================== 1 passed, 1 deselected in 0.1s =======================

@nprizal nprizal force-pushed the te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest branch from af0becc to 07121b3 Compare January 22, 2026 02:47
@nprizal nprizal force-pushed the te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest branch from 07121b3 to bf552fa Compare January 22, 2026 02:52
if plugin:
api = API(os.environ)
numprocesses = config.getoption("numprocesses")
numprocesses = config.getoption("numprocesses", None)
Copy link
Contributor Author

@nprizal nprizal Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the tests I added failed in CI because numprocesses option doesn't exists in the subprocess environment. I added None as default to prevent error being raised when evaluation that option.

@nprizal nprizal changed the title Add --tag-filters option to filter tests using execution_tag marker Add --tag-filters option to filter tests using execution_tag marker Jan 22, 2026
@nprizal nprizal self-assigned this Jan 22, 2026
@nprizal nprizal requested a review from gchan January 22, 2026 04:23
@nprizal nprizal marked this pull request as ready for review January 22, 2026 04:23
gchan added a commit to buildkite/test-engine-client that referenced this pull request Jan 23, 2026
Add a new --tag-filters CLI flag that allows filtering tests by execution
tags. This is useful for selectively running subsets of tests.

Currently only Pytest is supported via markers [1]. When specified, the
tag filter is passed to pytest via the -m flag (e.g., pytest -m
"test_execution").

We will replace the default marker filter with a custom filter option
implemented in the Python test collector [2]

Changes:
- Add --tag-filters flag and BUILDKITE_TEST_ENGINE_TAG_FILTERS env var
- Add TagFilters field to Config struct
- Update Pytest runner to use -m flag when tag filters are specified.
(This will be replaced with a custom filter)
- Add test coverage for marker-based filtering for pytest

Example usage:
  bktec run --tag-filters "slow"

  BUILDKITE_TEST_ENGINE_TAG_FILTERS="not integration" bktec run

[1] https://docs.pytest.org/en/7.1.x/example/markers.html
[2] buildkite/test-collector-python#83
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants