Add Sunbeam feature functional test suite

sunbeam-python/tests/functional/feature/.gitignore

@@ -0,0 +1,21 @@
+test_config.yaml
+features/adminrc
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+
+venv/
+env/
+ENV/
+
+.vscode/
+.idea/
+*.swp
+*.swo
+
+.pytest_cache/
+.coverage
+htmlcov/
+*.log

sunbeam-python/tests/functional/feature/README.md

@@ -0,0 +1,199 @@
+# Sunbeam Feature Functional Tests
+
+Functional tests for Sunbeam feature enablement/disablement. These tests
+connect to an **existing Sunbeam deployment** and run the enable/verify
+lifecycle for each feature, with an optional disable phase controlled by
+configuration.
+
+The suite is designed to be run via `tox` from the `sunbeam-python` tree.
+
+## Prerequisites
+
+- **Existing Sunbeam deployment** already bootstrapped and reachable
+- `sunbeam` CLI on `PATH` and configured to talk to that deployment
+  - e.g. `sunbeam deployment list` shows your deployment
+- `openstack` CLI configured for that cloud
+  - e.g. `openstack endpoint list` works
+- `juju` CLI installed and able to access the controller/model that backs the
+  Sunbeam deployment
+
+## Configuration
+
+Create a config file from the example:
+
+```bash
+cd sunbeam-python
+cp tests/functional/feature/test_config.yaml.example tests/functional/feature/test_config.yaml
+```
+
+Then edit `tests/functional/feature/test_config.yaml`:
+
+```yaml
+sunbeam:
+  deployment_name: "ps6"        # Name shown by `sunbeam deployment list`
+
+juju:
+  model: "openstack"            # Juju model backing the cloud
+  # controller: "my-controller" # Optional; auto-detected if omitted
+```
+
+### Run the full feature functional suite
+
+```bash
+tox -e functional-feature
+```
+
+### Run a single feature functional test
+
+You can pass standard `pytest` selectors through tox via `posargs`. For example:
+
+- **Instance Recovery**:
+
+  ```bash
+  tox -e functional-feature -- tests/functional/feature/test_features.py::test_instance_recovery
+  ```
+
+- **TLS CA**:
+
+  ```bash
+  tox -e functional-feature -- tests/functional/feature/test_features.py::test_tls_ca
+  ```
+
+- **Vault**:
+
+  ```bash
+  tox -e functional-feature -- tests/functional/feature/test_features.py::test_vault
+  ```
+
+You can also run any single feature test **directly with the virtualenv Python**,
+which is handy when you are iterating locally:
+
+```bash
+../.venv/bin/python -m pytest \
+  tests/functional/feature/test_features.py::test_<feature_name> \
+  --config tests/functional/feature/test_config.yaml
+```
+
+For example:
+
+- TLS CA:
+
+  ```bash
+  ../.venv/bin/python -m pytest \
+    tests/functional/feature/test_features.py::test_tls_ca \
+    --config tests/functional/feature/test_config.yaml
+  ```
+
+- Vault:
+
+  ```bash
+  ../.venv/bin/python -m pytest \
+    tests/functional/feature/test_features.py::test_vault \
+    --config tests/functional/feature/test_config.yaml
+  ```
+
+### Control whether features are disabled after tests
+
+By default, features are **left enabled** after their tests complete. You can
+enable the legacy "enable then disable" behaviour via `test_config.yaml`:
+
+```yaml
+features:
+  disable_after: true          # disable every feature after its test
+```
+
+You can also override this per feature:
+
+```yaml
+features:
+  disable_after: false         # default for all features
+
+  tls:
+    disable_after: true        # only TLS is disabled after test
+
+  vault:
+    disable_after: false       # explicitly keep Vault enabled
+
+You can also override this behaviour **from the command line** without editing
+the config file, using the `--features-disable-after` pytest option. When
+running via `tox`:
+
+```bash
+tox -e functional-feature -- --features-disable-after true   # force disable
+tox -e functional-feature -- --features-disable-after false  # force keep enabled
+```
+
+Or directly with the virtualenv Python:
+
+```bash
+../.venv/bin/python -m pytest \
+  tests/functional/feature/test_features.py::test_<feature_name> \
+  --config tests/functional/feature/test_config.yaml \
+  --features-disable-after true    # or false
+
+Concrete examples:
+
+- **Run TLS CA test and disable TLS after it completes**:
+
+  ```bash
+  tox -e functional-feature -- \
+    tests/functional/feature/test_features.py::test_tls_ca \
+    --features-disable-after true
+  ```
+
+- **Run Vault test and keep Vault enabled afterwards** (even if config sets
+  `disable_after: true`):
+
+  ```bash
+  tox -e functional-feature -- \
+    tests/functional/feature/test_features.py::test_vault \
+    --features-disable-after false
+  ```
+```
+
+## Feature coverage and dependencies
+
+### Features in this suite
+
+- **Enabled in current flow**
+  - `instance-recovery`
+  - `caas` (Containers as a Service)
+  - `dns`
+  - `images-sync`
+  - `loadbalancer`
+  - `resource-optimization`
+  - `shared-filesystem`
+  - `telemetry`
+  - `observability`
+  - `tls` (CA mode)
+  - `vault`
+  - `validation`
+  - `secrets`
+
+- **Present but intentionally disabled for now**
+  - `baremetal`
+  - `ldap`
+  - `maintenance`
+  - `pro`
+
+### Feature dependencies
+
+Some features have explicit dependencies:
+
+- **CaaS (`caas`)**
+  - Depends on: **`secrets`**, **`loadbalancer`**
+  - The CaaS test ensures these dependencies are enabled before running.
+
+- **Secrets as a Service (`secrets`)**
+  - Depends on: **`vault`**
+  - The Secrets test ensures the Vault feature is enabled before running.
+
+- **TLS (Vault-backed)**
+  - TLS can also be deployed in a Vault-backed mode which implicitly depends on
+    the **`vault`** feature. This suite currently exercises only the TLS CA
+    mode (`test_tls_ca`).
+
+## Notes
+
+- When `disable_after` is enabled (globally or per-feature), disable failures
+  are **logged and ignored** so that the suite continues to the next feature.

sunbeam-python/tests/functional/feature/__init__.py

@@ -0,0 +1,8 @@
+# SPDX-FileCopyrightText: 2024 - Canonical Ltd
+# SPDX-License-Identifier: Apache-2.0
+
+"""Sunbeam feature functional test suite.
+
+These tests exercise `sunbeam enable/disable` for individual features
+against an existing Sunbeam deployment.
+"""

sunbeam-python/tests/functional/feature/conftest.py

@@ -0,0 +1,88 @@
+# SPDX-FileCopyrightText: 2024 - Canonical Ltd
+# SPDX-License-Identifier: Apache-2.0
+
+"""Pytest configuration and fixtures for Sunbeam feature functional tests."""
+
+from pathlib import Path
+
+import pytest
+import yaml
+
+from .utils.juju import JujuClient
+from .utils.sunbeam import SunbeamClient
+
+
+def pytest_addoption(parser):
+    """Add custom command-line options."""
+    parser.addoption(
+        "--config",
+        action="store",
+        default="test_config.yaml",
+        help="Path to test configuration file",
+    )
+    parser.addoption(
+        "--features-disable-after",
+        action="store",
+        choices=["true", "false"],
+        default=None,
+        help=(
+            "Override features.disable_after (true/false) from test_config.yaml "
+            "without editing the file."
+        ),
+    )
+
+
+@pytest.fixture(scope="session")
+def test_config(request):
+    """Load test configuration from YAML file."""
+    config_path = request.config.getoption("--config")
+    # Resolve relative to this feature functional directory
+    config_file = Path(__file__).parent / config_path
+
+    if not config_file.exists():
+        msg = (
+            f"Configuration file not found: {config_file}. "
+            "Copy tests/functional/feature/test_config.yaml.example to "
+            "tests/functional/feature/test_config.yaml and set sunbeam.deployment_name, juju.model."
+        )
+        pytest.skip(msg)
+
+    with open(config_file, "r") as f:
+        config = yaml.safe_load(f)
+
+    # Optional CLI override for disable-after behaviour.
+    cli_disable_after = request.config.getoption("--features-disable-after")
+    if cli_disable_after is not None:
+        features_cfg = config.setdefault("features", {})
+        features_cfg["disable_after"] = cli_disable_after == "true"
+
+    return config
+
+
+@pytest.fixture(scope="session")
+def sunbeam_client(test_config):
+    """Create Sunbeam client for test session."""
+    deployment_name = test_config.get("sunbeam", {}).get("deployment_name")
+    if not deployment_name:
+        pytest.skip("deployment_name not configured in test_config.yaml")
+
+    client = SunbeamClient(deployment_name)
+
+    if not client.is_connected():
+        pytest.skip(f"Cannot connect to Sunbeam deployment '{deployment_name}'.")
+
+    return client
+
+
+@pytest.fixture(scope="session")
+def juju_client(test_config):
+    """Create Juju client for test session."""
+    model = test_config.get("juju", {}).get("model", "openstack")
+    controller = test_config.get("juju", {}).get("controller")
+
+    client = JujuClient(model=model, controller=controller)
+
+    if not client.is_connected():
+        pytest.skip(f"Cannot connect to Juju model '{model}'.")
+
+    return client

sunbeam-python/tests/functional/feature/features/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-FileCopyrightText: 2024 - Canonical Ltd
+# SPDX-License-Identifier: Apache-2.0
+
+"""Feature test classes for Sunbeam feature functional tests."""

sunbeam-python/tests/functional/feature/features/baremetal.py

@@ -0,0 +1,42 @@
+# SPDX-FileCopyrightText: 2024 - Canonical Ltd
+# SPDX-License-Identifier: Apache-2.0
+
+"""Test for baremetal feature.
+
+Baremetal provides Ironic-based bare metal provisioning.
+Functionality is validated via the Ironic (baremetal) API.
+"""
+
+import logging
+import subprocess
+
+from .base import BaseFeatureTest
+
+logger = logging.getLogger(__name__)
+
+
+class BaremetalTest(BaseFeatureTest):
+    """Test baremetal feature enablement/disablement."""
+
+    feature_name = "baremetal"
+    expected_applications: list[str] = []
+    timeout_seconds = 600
+
+    def verify_validate_feature_behavior(self) -> None:
+        """Validate that the Baremetal (Ironic) API is reachable."""
+        logger.info("Verifying Baremetal (Ironic) service is available...")
+        try:
+            subprocess.run(
+                ["openstack", "baremetal", "driver", "list"],
+                capture_output=True,
+                text=True,
+                timeout=30,
+                check=True,
+            )
+        except Exception as exc:  # noqa: BLE001
+            logger.warning("Error while verifying Baremetal service: %s", exc)
+            raise AssertionError(
+                f"Baremetal service verification failed: {exc}"
+            ) from exc
+
+        logger.info("Baremetal service verified via `openstack baremetal driver list`")