feat(storage): Add external storage backend integration via Sunbeam CLI

[happyhackerhour]

Sunbeam Storage Backend Management

Summary

This PR introduces a comprehensive storage backend management system for Sunbeam with full unit testing coverage. It provides a clean, maintainable framework for managing Cinder storage backends, starting with Hitachi VSP support, featuring separated CLI architecture, dynamic configuration management, and 1029+ tests ensuring reliability.

Context & Related Work

Builds on:

• cinder-volume-hitachi charm https://review.opendev.org/c/openstack/sunbeam-charms/+/955633 - Configures Hitachi VSP drivers
• *snap-cinder-volume https://github.com/happyhackerhour/snap-cinder-volume/tree/feat/hitachi-backend - Backend configuration rendering
• This PR - Sunbeam CLI and management framework

Key Changes

1. Storage Backend Framework (sunbeam/storage/)

• base.py - Core service layer with StorageBackendService and proper CLI registration hooks
• registry.py - Dynamic backend discovery and CLI registration with plugin architecture
• service.py - Centralized Terraform state management and backend existence checking
• steps.py - Deployment workflow with comprehensive error handling
• Exception hierarchy for robust error handling and user feedback

2. Hitachi VSP Backend (sunbeam/storage/backends/hitachi/)

• backend.py - Core deployment and management logic with HitachiConfig Pydantic model
• cli.py - Dedicated CLI commands with proper separation of concerns
• 6 essential deployment fields with comprehensive validation
• IP/FQDN validation with proper error handling
• Protocol selection (FC/iSCSI) with validation
• Configuration filtering logic to prevent sensitive data exposure

3. Dynamic Configuration Management

• get_backend_config() - Retrieve current configuration with filtering
• CLI-based configuration management with rich console integration
• Sensitive value masking for security (passwords, secrets)
• Service layer delegation for all Terraform state operations
• Validation and error handling for configuration updates

CLI Commands

Backend Management

# Deploy backend
sunbeam storage add <backend>                              # Interactive deployment
sunbeam storage add <backend> --name <name> [options]     # Non-interactive deployment
sunbeam storage remove <backend> <name>                   # Remove existing backend  
sunbeam storage list all                                # List all backends

Configuration Management

# Configuration operations
sunbeam storage config show <backend> <name>              # View current configuration
sunbeam storage config set <backend> <name> [options]           # Update configuration
sunbeam storage config reset <backend> <name> [keys]     # Reset specific keys
sunbeam storage config options <backend>                 # List available options

Example Usage

Deploy Backend

Interactive deployment:

$ sunbeam storage add hitachi
SAN Management IP: 192.168.1.100
SAN Username: maintenance  
Storage Pools: pool1,pool2
Protocol: FC

✅ Successfully deployed Hitachi backend 'production-hitachi'

Direct deployment with parameters:

$ sunbeam storage add hitachi --name vsp350 \
    --san-ip=192.168.1.100 \
    --san-username=maintenance \
    --san-password=12345 \
    --hitachi-pools=pool1,pool2 \
    --protocol=FC \
    --hitachi-storage-id=123123123

Manage Configuration

View configuration:

$ sunbeam storage config show hitachi vsp350
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Configuration Key  ┃ Value              ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ san-ip             │ 192.168.1.100      │
│ san-login          │ maintenance        │
│ hitachi-pools      │ pool1,pool2        │
│ protocol           │ FC                 │
│ san-password       │ ********           │
└────────────────────┴────────────────────┘

Update configuration:

$ sunbeam storage config set hitachi vsp350 --hitachi-copy-speed=3
✅ Configuration updated successfully

List available options:

$ sunbeam storage config options hitachi
Available configuration options for Hitachi VSP Storage:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Option                       ┃ Description                                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ san-ip                       │ SAN management IP address or FQDN                │
│ san-login                    │ SAN management username                          │
│ san-password                 │ SAN management password                          │
│ hitachi-pools                │ Comma-separated list of storage pools            │
│ protocol                     │ Storage protocol (FC or iSCSI)                   │
│ hitachi-copy-speed           │ Copy speed for volume operations (1-15)          │
└──────────────────────────────┴───────────────────────────────────────────────────┘

[wolsen]

I'm currently making some inflight comments. I have not had a chance to perform an exhaustive review.

[gboutry]

My main comment was lost with that whole shebang of pending review...

This is overall a nice work, and thank you very much for your contribution.

Can you ensure this is inline with the rest of the code base?
Can you reduce the use of emojis?
Can you format the code? There are a lot of trailing whitespaces.

[Copilot]

Pull Request Overview

This PR introduces a comprehensive storage backend management system for Sunbeam, featuring Terraform-managed Hitachi VSP support, dynamic backend discovery via a pluggable registry, and robust unit testing coverage with 109 tests. The implementation provides CLI commands for storage backend deployment, configuration management, and lifecycle operations.

Key changes:

• Storage backend framework with base classes, service layer, and registry system for dynamic backend discovery
• Hitachi VSP backend implementation with complete Terraform deployment and credential management via Juju secrets
• CLI integration providing commands for add, remove, list, and config operations with both interactive and inline configuration modes

Reviewed Changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sunbeam/storage/base.py	Base classes and interfaces for storage backend implementations with Terraform integration
sunbeam/storage/backends/hitachi/backend.py	Complete Hitachi VSP backend implementation with configuration model and deployment steps
sunbeam/storage/backends/hitachi/deploy-hitachi-backend/main.tf	Terraform configuration for Hitachi backend deployment with Juju secrets management
sunbeam/storage/registry.py	Dynamic backend discovery and CLI command registration system
sunbeam/storage/service.py	Service layer for backend operations and configuration management
sunbeam/storage/steps.py	Base step classes for deployment, destruction, and configuration update workflows
sunbeam/commands/storage.py	CLI commands for storage backend management
tests/unit/sunbeam/storage/	Comprehensive unit test suite with 109 tests covering all modules

Comments suppressed due to low confidence (2)

sunbeam-python/sunbeam/storage/steps.py:24

• Import error: BackendNotFoundException is imported from .models but according to the models.py file, this exception should be imported from .models which is correct. However, there's also an import from sunbeam.storage.models on line 25 which creates a redundant import structure. The imports should be consolidated.

from sunbeam.storage.models import BackendNotFoundException

sunbeam-python/sunbeam/storage/backends/hitachi/deploy-hitachi-backend/main.tf:8

• The Juju Terraform provider version "= 0.20.0" may not exist or may be outdated. Consider using a more recent or verified version, or use a version constraint like "~> 0.20" to allow patch updates.

      version = "= 0.20.0"

[happyhackerhour]

Hi team,

I’ve finished the requested fixes and rewrites, doing my best to align with the current style and patterns across the project.

I appreciate your time and effort in reviewing this work!