Add Sunbeam feature functional test suite by ahmad-can · Pull Request #652 · canonical/snap-openstack

ahmad-can · 2026-01-27T13:54:25Z

No description provided.

hemanthnakkina · 2026-01-27T15:33:20Z

Can we think about replacing https://github.com/canonical/snap-openstack/blob/main/.github/workflows/build-snap.yml#L94 with these functional tests

Copilot

Pull request overview

Adds a new “feature functional” test suite intended to run against an existing Sunbeam deployment, with tox wiring and per-feature lifecycle tests (enable/verify/optional disable).

Changes:

Add functional-feature tox environment to execute the new feature functional test suite.
Introduce pytest-based feature tests plus CLI wrappers (Sunbeam + Juju) and a shared BaseFeatureTest.
Add suite documentation, example configuration, and local ignores for config artifacts.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
sunbeam-python/tox.ini	Adds `functional-feature` tox env to run the new suite.
sunbeam-python/tests/functional/feature/init.py	Declares the feature functional test package.
sunbeam-python/tests/functional/feature/.gitignore	Ignores local config/adminrc artifacts for the suite.
sunbeam-python/tests/functional/feature/README.md	Documents prerequisites, config, and how to run the suite.
sunbeam-python/tests/functional/feature/conftest.py	Adds pytest options and session fixtures for config + clients.
sunbeam-python/tests/functional/feature/pytest.ini	Suite-specific pytest settings (markers/opts/timeout).
sunbeam-python/tests/functional/feature/requirements.txt	Lists Python deps for running the suite standalone.
sunbeam-python/tests/functional/feature/test_config.yaml.example	Provides an example config for running against a deployment.
sunbeam-python/tests/functional/feature/test_features.py	Orchestrates running per-feature test classes.
sunbeam-python/tests/functional/feature/utils/init.py	Marks utility wrapper package.
sunbeam-python/tests/functional/feature/utils/sunbeam.py	Adds a `SunbeamClient` wrapper around the CLI.
sunbeam-python/tests/functional/feature/utils/juju.py	Adds a `JujuClient` wrapper using Jubilant.
sunbeam-python/tests/functional/feature/features/init.py	Marks feature test classes package.
sunbeam-python/tests/functional/feature/features/base.py	Introduces shared lifecycle logic and OpenStack env setup.
sunbeam-python/tests/functional/feature/features/baremetal.py	Adds baremetal feature lifecycle checks (currently skipped by runner).
sunbeam-python/tests/functional/feature/features/caas.py	Adds caas feature lifecycle checks + dependency enablement.
sunbeam-python/tests/functional/feature/features/dns.py	Adds dns feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/images_sync.py	Adds images-sync feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/instance_recovery.py	Adds instance-recovery feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/ldap.py	Adds ldap feature lifecycle checks (currently skipped by runner).
sunbeam-python/tests/functional/feature/features/loadbalancer.py	Adds loadbalancer feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/maintenance.py	Adds maintenance feature lifecycle checks (currently skipped by runner).
sunbeam-python/tests/functional/feature/features/observability.py	Adds observability embedded workflow checks.
sunbeam-python/tests/functional/feature/features/orchestration.py	Adds orchestration (heat) feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/pro.py	Adds pro feature lifecycle checks (currently skipped by runner).
sunbeam-python/tests/functional/feature/features/resource_optimization.py	Adds resource-optimization feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/secrets.py	Adds secrets feature lifecycle checks + vault prerequisite.
sunbeam-python/tests/functional/feature/features/shared_filesystem.py	Adds shared-filesystem (manila) feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/telemetry.py	Adds telemetry feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/tls.py	Adds TLS CA + TLS Vault test flows, CSR signing, and endpoint validation.
sunbeam-python/tests/functional/feature/features/validation.py	Adds validation feature lifecycle checks.
sunbeam-python/tests/functional/feature/features/vault.py	Adds vault enable/init/unseal/authorize flow and verification.

Comments suppressed due to low confidence (1)

sunbeam-python/tests/functional/feature/features/base.py:262

_ensure_openstack_env ends after logging an exception when writing/loading the generated openrc fails, and there is no fallback path when sunbeam openrc fails. This can leave OpenStack env vars unset and make all subsequent openstack CLI checks fail; implement the documented fallback (e.g., load an existing adminrc) and/or return early with a clear skip/error when credentials can't be loaded.

                except Exception:  # noqa: BLE001
                    logger.exception(
                        "Failed to write or load openrc from %s", adminrc_path
                    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-24T12:40:05Z

+```bash
+../.venv/bin/python -m pytest \
+  tests/functional/feature/test_features.py::test_<feature_name> \
+  --config tests/functional/feature/test_config.yaml
+```


The README examples pass --config tests/functional/feature/test_config.yaml, but conftest.py currently resolves --config relative to tests/functional/feature/, which will duplicate the path and skip. Either update the examples to use --config test_config.yaml (or adjust config path resolution in conftest.py to accept repo-relative paths).

Copilot · 2026-02-24T12:40:06Z

+juju:
+  # Juju model name (default: "openstack")
+  model: "openstack"
+  # Juju controller (auto-detected from sunbeam if not specified)


This comment says the Juju controller is auto-detected from sunbeam when omitted, but the current JujuClient implementation doesn't do any controller detection/switching. Update the comment (or implement controller selection) so users aren't misled.

Suggested change

# Juju controller (auto-detected from sunbeam if not specified)

# Juju controller name (uncomment and set to target a specific controller)

Copilot · 2026-02-24T12:40:06Z

+  vault:
+    disable_after: false       # explicitly keep Vault enabled
+
+You can also override this behaviour **from the command line** without editing
+the config file, using the `--features-disable-after` pytest option. When


The fenced YAML block starting at You can also override this per feature: is missing a closing ``` before the following paragraph, so the rest of the document renders as code. Close the YAML fence after the vault: example.

Copilot · 2026-02-24T12:40:06Z

+pytest>=7.4.0
+pytest-timeout>=2.1.0
+pyyaml>=6.0
+jubilant>=1.0.0


jubilant>=1.0.0 here conflicts with the project dependency in pyproject.toml (currently jubilant>=1.3.0). Align the minimum versions (or explain why this requirements file intentionally differs) to avoid confusing or broken standalone installs of the feature test deps.

Suggested change

jubilant>=1.0.0

jubilant>=1.3.0

Copilot · 2026-02-24T12:40:07Z

+timeout = 1800
+


timeout = 1800 requires the pytest-timeout plugin; it isn't included in the project's dev extra (and doesn't appear in uv.lock). If this pytest.ini is meant to be used in tox, either add pytest-timeout to the environment deps/dev extra or remove the timeout setting to avoid pytest failing with an unknown config option.

Suggested change

timeout = 1800

Copilot · 2026-02-24T12:40:08Z

+commands = uv run {[vars]uv_flags} \
+  python -m pytest -s -vv tests/functional/feature \
+  --config=test_config.yaml \
+  --basetemp={env:HOME}/.local/share/openstack/tmp \
+  {posargs}


tests/functional/feature/pytest.ini won't be picked up when running pytest from the repository root (pytest will use the top-level pyproject.toml config instead). If you want these marker/timeouts/settings applied for the tox env, pass -c tests/functional/feature/pytest.ini (or move the relevant options to the project-level pytest config).

Copilot · 2026-02-24T12:40:09Z

+        result = subprocess.run(cmd, check=True, capture_output=True, text=True)
+        if result.returncode != 0:
+            raise AssertionError(
+                "Basic OpenStack operation failed after TLS Vault enablement: "
+                f"openstack image list returned error: {result.stderr}"
+            )


This returncode check is redundant/unreachable because subprocess.run(..., check=True) will raise on non-zero exit codes. Either drop the if result.returncode != 0 block or switch to check=False and handle stderr explicitly.

Suggested change

result = subprocess.run(cmd, check=True, capture_output=True, text=True)

if result.returncode != 0:

raise AssertionError(

"Basic OpenStack operation failed after TLS Vault enablement: "

f"openstack image list returned error: {result.stderr}"

)

subprocess.run(cmd, check=True, capture_output=True, text=True)

Copilot · 2026-02-24T12:40:10Z

+    def is_connected(self) -> bool:
+        """Check if we can connect to the Sunbeam deployment."""
+        result = subprocess.run(
+            ["sunbeam", "deployment", "list"],
+            capture_output=True,


is_connected() hardcodes sunbeam from PATH, while the rest of this wrapper uses self._sunbeam_cmd ("/snap/bin/sunbeam"). Use the same command path here (or consistently resolve via PATH) so connectivity checks behave the same as enable/disable/run.

Copilot · 2026-02-24T12:40:10Z

+```bash
+../.venv/bin/python -m pytest \
+  tests/functional/feature/test_features.py::test_<feature_name> \
+  --config tests/functional/feature/test_config.yaml \
+  --features-disable-after true    # or false


The fenced bash block starting at Or directly with the virtualenv Python: isn't closed until much later, which pulls headings and prose into the code block. Add a closing ``` right after the command example (after the --features-disable-after ... line).

Copilot · 2026-02-24T12:40:10Z

+        ]
+
+        self.enable()
+
+        # Automate the external CA flow: list CSRs, sign with our CA, and
+        # provide the signed certificate back to Vault.
+        self._sign_vault_csrs_and_install()


run_full_lifecycle() ignores the return value of self.enable(). If enablement fails, the subsequent CSR signing and validation steps will run anyway and fail in less clear ways; return early (or raise) when enable() returns False.

hemanthnakkina · 2026-05-28T03:11:21Z

@ahmad-can Can you abandon this PR as you have already moved the functional tests to right repo

ahmad-can force-pushed the sunbeam-features-functional-tests branch from b11a6cf to da340ba Compare January 27, 2026 14:38

Add Sunbeam feature functional test suite

fabe182

ahmad-can force-pushed the sunbeam-features-functional-tests branch from da340ba to fabe182 Compare January 27, 2026 14:48

Refine Sunbeam feature functional tests

0d8b6d0

ahmad-can force-pushed the sunbeam-features-functional-tests branch from 7d1ed04 to 7e6cc1e Compare February 3, 2026 10:52

ahmad-can mentioned this pull request Feb 17, 2026

Sunbeam chaos tests #667

Open

Unify feature test lifecycle

7561df2

ahmad-can force-pushed the sunbeam-features-functional-tests branch from 6a292be to 7561df2 Compare February 18, 2026 11:34

Refine TLS feature tests

5905c21

ahmad-can force-pushed the sunbeam-features-functional-tests branch from bab2a3b to 5905c21 Compare February 23, 2026 06:54

load OpenStack env via sunbeam openrc

b490f6d

hemanthnakkina requested a review from Copilot February 24, 2026 12:33

Copilot started reviewing on behalf of hemanthnakkina February 24, 2026 12:34 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

ahmad-can added 2 commits February 26, 2026 11:08

fix tls ca

5351795

Update DNS feature tests to align CLI usage and fix lint

0b375a0

ahmad-can force-pushed the sunbeam-features-functional-tests branch from a459484 to 0b375a0 Compare March 3, 2026 08:38

	# Juju controller (auto-detected from sunbeam if not specified)
	# Juju controller name (uncomment and set to target a specific controller)

Conversation

ahmad-can commented Jan 27, 2026

Uh oh!

hemanthnakkina commented Jan 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

hemanthnakkina commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants