Skip to content

ci: add GitHub Actions workflow to run tests on PRs #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mrshu wants to merge 1 commit into from
Closed

ci: add GitHub Actions workflow to run tests on PRs #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Tests

on:
pull_request:
push:
branches: [main]

jobs:
core:
name: Core tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0

- name: Install core dependencies
run: uv sync --locked

- name: Run core tests
run: |
uv run pytest \
tests/test_validate.py \
tests/test_check_duplicate_entries.py \
tests/test_inspect_uuid_utils.py \
tests/test_cli_inspect_uuid.py \
tests/test_lm_eval_adapter.py \
-v

inspect:
name: Inspect converter tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0

- name: Install dependencies with inspect extra
run: uv sync --locked --extra inspect

- name: Run inspect tests
run: |
uv run pytest \
tests/test_inspect_adapter.py \
tests/test_inspect_instance_level_adapter.py \
-v

helm:
name: HELM converter tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0

- name: Install dependencies with helm extra
run: uv sync --locked --extra helm

- name: Run HELM tests
run: |
uv run pytest \
tests/test_helm_adapter.py \
tests/test_helm_instance_level_adapter.py \
-v
Comment on lines +1 to +64
Copy link
Copy Markdown
Collaborator

@Erotemic Erotemic Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: Tests
on:
pull_request:
push:
branches: [main]
jobs:
core:
name: Core tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0
- name: Install core dependencies
run: uv sync --locked
- name: Run core tests
run: |
uv run pytest \
tests/test_validate.py \
tests/test_check_duplicate_entries.py \
tests/test_inspect_uuid_utils.py \
tests/test_cli_inspect_uuid.py \
tests/test_lm_eval_adapter.py \
-v
inspect:
name: Inspect converter tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0
- name: Install dependencies with inspect extra
run: uv sync --locked --extra inspect
- name: Run inspect tests
run: |
uv run pytest \
tests/test_inspect_adapter.py \
tests/test_inspect_instance_level_adapter.py \
-v
helm:
name: HELM converter tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0
- name: Install dependencies with helm extra
run: uv sync --locked --extra helm
- name: Run HELM tests
run: |
uv run pytest \
tests/test_helm_adapter.py \
tests/test_helm_instance_level_adapter.py \
-v
name: Tests
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
tests:
name: ${{ matrix.deps }} / ${{ matrix.resolution }}
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
include:
- deps: core
resolution: locked
sync: uv sync --locked
pytest_args: >-
tests
--ignore-glob=tests/test_inspect*.py
--ignore-glob=tests/test_helm*.py
-v
- deps: core
resolution: loose
sync: uv sync --upgrade
pytest_args: >-
tests
--ignore-glob=tests/test_inspect*.py
--ignore-glob=tests/test_helm*.py
-v
- deps: full
resolution: locked
sync: uv sync --locked --all-extras
pytest_args: tests -v
- deps: full
resolution: loose
sync: uv sync --upgrade --all-extras
pytest_args: tests -v
steps:
- uses: actions/checkout@v6.0.2
- uses: astral-sh/setup-uv@v7.6.0
- name: Install dependencies
run: ${{ matrix.sync }}
- name: Run tests
run: uv run pytest ${{ matrix.pytest_args }}

Copy link
Copy Markdown
Collaborator

@Erotemic Erotemic Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the style of CI that I like to use. I also like to use a "core" or "minimal" test of tests that just target the plain library without any extras. When dealing with extras instead of worrying about diffrent combinations I like to use a "full" version where all the extras are installed. We could split out the inspect / helm versions but I think it's not worth the effort.

What I do think is worth the effort is a locked / loose install check. I find it extremely valuable to test the locked version of the deps as well as the deps that would be installed if a user pip installed today. This gives a strong signal when an upstream package causes a break in this package.

For now I think it's fine to use the --ignore-glob, but really we should just use pytest to mark which tests should / shouldn't run based on the installed packages, but the glob method is good enough for now.

36 changes: 36 additions & 0 deletions uv.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading