Eai 1452 seaweed s3 rebased 4

PRD.md

@@ -61,22 +61,22 @@ Three cluster profiles with inheritance-based resource optimization:
 - Single replica deployments (ArgoCD, Redis, etc.)
 - Reduced resource limits (ArgoCD controller: 2 CPU, 2Gi RAM)
 - Adds kyverno-policies-storage-local-path for RWX→RWO PVC mutation
-- MinIO tenant: 2Ti storage, single server
+- SeaweedFS volume storage: 2Ti, single volume server
 - Mix of local-path and direct storage classes
 - Suitable for: Local workstations, development environments
 
 **Medium Clusters** (5-20 users, team production):
 - Single replica with moderate resource allocation
 - Same storage policies as small (local-path support)
 - ArgoCD controller: 1 CPU, 2Gi RAM
-- MinIO tenant: 2Ti storage
+- SeaweedFS volume storage: 2Ti
 - Uses direct storage class consistently
 - Suitable for: Small teams, staging environments
 
 **Large Clusters** (10s-100s users, enterprise scale):
 - OpenBao HA: 3 replicas with Raft consensus
 - No local-path policies (assumes distributed storage like Longhorn)
-- MinIO tenant: 500Gi storage
+- SeaweedFS volume storage: 500Gi
 - Production-grade resource allocation
 - Uses direct storage class for all persistent volumes
 - Suitable for: Production deployments, multi-tenant environments
@@ -130,8 +130,8 @@ The cluster-forge Application uses multi-source feature when externalValues.enab
 
 *Storage & Database:*
 - CNPG Operator 0.26.0 - CloudNativePG PostgreSQL operator
-- MinIO Operator 7.1.1 - S3-compatible object storage operator
-- MinIO Tenant 7.1.1 - Tenant deployment with default-bucket and models buckets
+- SeaweedFS Operator - S3-compatible object storage operator
+- SeaweedFS Config - S3 storage deployment with default-bucket, models, and datasets buckets
 
 **Layer 3: Observability** (Sync Wave -5 to -2)
 - Prometheus Operator CRDs 23.0.0 - Metrics infrastructure
@@ -145,7 +145,8 @@ The cluster-forge Application uses multi-source feature when externalValues.enab
 - Keycloak (keycloak-old chart) - Enterprise IAM with AIRM realm
   - Custom extensions via init containers (SilogenExtensionPackage.jar)
   - Realm import with domain-group-authenticator
-  - Client secrets for: AIRM, K8s, MinIO, Gitea, ArgoCD
+  - Client secrets for: AIRM, K8s, S3 (SeaweedFS), Gitea, ArgoCD
+  - Note: S3 credentials in OpenBao still use "minio-*" paths for backward compatibility
 
 **Layer 5: AI/ML Compute Stack** (Sync Wave -3 to 0)
 
@@ -350,7 +351,7 @@ Each major component has -config variant:
 - argocd-config: OIDC integration, RBAC policies, ExternalSecrets
 - gitea-config: Keycloak OAuth, repository templates
 - openbao-config: Policy definitions, secret paths, initialization scripts
-- minio-tenant-config: Bucket policies, user credentials, gateway routes
+- seaweedfs-config: S3 configuration, user credentials, gateway routes, bucket initialization
 
 ### Secrets Management Architecture
 
@@ -413,7 +414,7 @@ scripts/utils/import_rabbitmq.sh    # Restore queues and exchanges
 
 **Object Storage:**
 ```bash
-scripts/utils/mirror_minio.sh       # MinIO bucket synchronization
+scripts/utils/mirror-minio-to-seaweedfs-job.yaml  # MinIO to SeaweedFS migration job
 ```
 
 ### Observability Stack
@@ -458,7 +459,7 @@ Kueue manages scheduling for:
 - KServe Standard deployment mode
 - InferenceService CRD for models
 - Auto-scaling with KEDA
-- S3 model storage via MinIO
+- S3 model storage via SeaweedFS
 
 **GPU Support:**
 - AMD GPU Operator for device plugin

README.md

@@ -104,8 +104,8 @@ See [Values Inheritance Pattern](docs/values_inheritance_pattern.md) for detaile
 
 **Storage & Database:**
 - **CNPG Operator 0.26.0** - CloudNativePG PostgreSQL operator
-- **MinIO Operator 7.1.1** - S3-compatible object storage operator
-- **MinIO Tenant 7.1.1** - Tenant deployment with default-bucket and models buckets
+- **SeaweedFS Operator** - S3-compatible object storage operator
+- **SeaweedFS Config** - S3 storage deployment with default-bucket, models, and datasets buckets
 
 ### Layer 3: Observability
 - **Prometheus Operator CRDs 23.0.0** - Metrics infrastructure
@@ -147,7 +147,7 @@ Three cluster profiles with inheritance-based resource optimization:
 - Single replica deployments
 - Reduced resource limits (ArgoCD controller: 2 CPU, 4Gi RAM)
 - Adds kyverno-policies-storage-local-path for RWX→RWO PVC mutation
-- MinIO tenant: 250Gi storage
+- SeaweedFS volume storage: 250Gi
 - Suitable for: Local workstations, development environments
 
 **Medium Clusters** (5-20 users, team production):
@@ -162,7 +162,7 @@ Three cluster profiles with inheritance-based resource optimization:
 - Requires a minimum of 20 CPU cores
 - OpenBao HA: 3 replicas with Raft consensus
 - No local-path policies (assumes distributed storage)
-- MinIO tenant: 500Gi storage
+- SeaweedFS volume storage: 500Gi
 - Production-grade resource allocation
 - Suitable for: Production deployments, multi-tenant environments
 

docs/backup_and_restore.md

@@ -10,7 +10,7 @@
 This document covers backup and restore procedures for:
   1. [Database Backup & Restore (AMD Resource Manager & Keycloak)](#1-cnpg-cloudnative-postgres-backup--restore)
   2. [RabbitMQ Backup & Restore](#2-rabbitmq-backup--restore)
-  3. [MinIO Backup & Restore (Bucket replication and one-off filesystem backup)](#3-minio-backup--restore)
+  3. [SeaweedFS Backup & Restore (S3-compatible object storage - bucket replication and one-off filesystem backup)](#3-seaweedfs-backup--restore)
   4. [Longhorn Backup & Restore](#4-longhorn-backup--restore)
 
 **Note:** Each detailed command file includes specific prerequisites for that backup type.
@@ -74,11 +74,11 @@ For a backup & restore example (not officially supported), see: [`backup_restore
 
 ---
 
-## 3. MinIO Backup & Restore
+## 3. SeaweedFS Backup & Restore
 
 ### Overview
 
-MinIO supports two backup strategies:
+SeaweedFS S3-compatible storage supports two backup strategies:
 
 #### Two-Way Replication (Bucket Replication)
 **Note:** Bucket replication and site replication are mutually exclusive.
@@ -95,13 +95,13 @@ MinIO supports two backup strategies:
 2. Create a timestamped backup directory
 3. Use `mc mirror` to copy all bucket contents to the backup location
 4. Provide verification commands to compare file counts
-5. Provide restore commands to mirror files back to MinIO
+5. Provide restore commands to mirror files back to SeaweedFS
 
 **Benefits:** Point-in-time snapshots stored on separate storage
 
 ### Detailed Commands
 
-For a backup & restore example (not officially supported), see:: [`backup_restore_minio.md`](examples/backup_restore_minio.md)
+For a backup & restore example (not officially supported), see: [`backup_restore_seaweedfs.md`](examples/backup_restore_seaweedfs.md)
 
 ---
 

docs/cluster_size_configuration.md

@@ -40,7 +40,7 @@ cluster-forge/
 
 **Application Configuration**:
 - **ArgoCD**: Single replica, 2 CPU / 2Gi RAM limits
-- **MinIO Tenant**: 2Ti storage, single server
+- **SeaweedFS**: 2Ti volume storage, single volume server
 - **OpenBao**: Single instance (no HA), 5Gi storage
 - **Storage Policies**: Includes `kyverno-policies-storage-local-path` for RWX→RWO conversion
 - **Storage Classes**: Mix of local-path and direct storage classes
@@ -61,7 +61,7 @@ cluster-forge/
 
 **Application Configuration**:
 - **ArgoCD**: Single replica, 1 CPU / 2Gi RAM limits
-- **MinIO Tenant**: 2Ti storage, single server
+- **SeaweedFS**: 2Ti volume storage, single volume server
 - **OpenBao**: Single instance (no HA), 5Gi storage
 - **Storage Policies**: Includes `kyverno-policies-storage-local-path` for RWX→RWO conversion
 - **Storage Classes**: Direct storage class consistently
@@ -82,7 +82,7 @@ cluster-forge/
 
 **Application Configuration**:
 - **ArgoCD**: Single replica, production-ready resources
-- **MinIO Tenant**: 500Gi storage, single server (external HA S3 recommended)
+- **SeaweedFS**: 500Gi volume storage, single volume server (external HA S3 recommended)
 - **OpenBao**: 3 replicas with Raft HA consensus
 - **Storage Policies**: No local-path policies (assumes distributed storage)
 - **OTEL LGTM Stack**: 50Gi storage per component (Tempo, Loki, Mimir), 10Gi Grafana
@@ -149,13 +149,13 @@ Later values override earlier ones, allowing size files to contain only the diff
 | KEDA | Base config | Base config | Base config | Event-driven autoscaling |
 | KServe | Base config | Base config | Base config | ML model serving |
 | Kyverno | Base policies | Base + storage-local-path | Base policies only | Policy engine |
-### MinIO Tenant Scaling
+### SeaweedFS Scaling
 
-| Size | Servers | Storage | Buckets | Notes |
-|------|---------|---------|---------|-------|
-| Small | 1 | 2Ti | default-bucket, models | Single server, local-path storage |
-| Medium | 1 | 2Ti | default-bucket, models | Single server, direct storage |
-| Large | 1 | 500Gi | default-bucket, models | Single server, external HA S3 recommended |
+| Size | Volume Servers | Storage | Buckets | Notes |
+|------|----------------|---------|---------|-------|
+| Small | 1 | 2Ti | default-bucket, models, datasets | Single volume server, local-path storage |
+| Medium | 1 | 2Ti | default-bucket, models, datasets | Single volume server, direct storage |
+| Large | 1 | 500Gi | default-bucket, models, datasets | Single volume server, external HA S3 recommended |
 
 ### OpenBao Scaling
 

docs/examples/backup_restore_minio.md → docs/examples/backup_restore_seaweedfs.md

@@ -1,6 +1,6 @@
-# MinIO Backup & Restore Commands
+# SeaweedFS Backup & Restore Commands
 
-This document provides the step-by-step commands for MinIO backup and restore operations.
+This document provides the step-by-step commands for SeaweedFS backup and restore operations.
 
 ⚠️ Important Disclaimers
   - This is only an example script only, adjust paths and commands as needed for your system.
@@ -12,11 +12,11 @@ This document provides the step-by-step commands for MinIO backup and restore op
 ## Prerequisites
 
 - Shell access to a machine with `kubectl` configured for the target Kubernetes cluster
-- MinIO client (`mc`) installed on your local machine, **OR** access to run commands inside the `minio` pod in the MinIO tenant namespace
+- MinIO client (`mc`) installed on your local machine (SeaweedFS S3 API is compatible with `mc`)
   - To install `mc` locally: [https://min.io/docs/minio/linux/reference/minio-mc.html](https://min.io/docs/minio/linux/reference/minio-mc.html)
-  - To run inside the pod: `kubectl exec -it -n <minio-namespace> <minio-pod-name> -- sh`
+  - The MinIO client works with any S3-compatible storage, including SeaweedFS
 - For NFS backups: Access to mount NFS shares (requires `sudo` privileges)
-- For bucket replication: Access to both source and destination MinIO endpoints
+- For bucket replication: Access to both source and destination S3-compatible endpoints
 - The `jq` command-line JSON processor (for replication resync commands)
 
 ## 1. Setup Two-Way Replication (Bucket replication)
@@ -25,15 +25,16 @@ This document provides the step-by-step commands for MinIO backup and restore op
 
 These commands are run from a local machine with MinIO client (`mc`) installed.
 
-### Step 1: Configure MinIO Aliases
+### Step 1: Configure S3 Aliases
 ```bash
-# Set up source and destination MinIO endpoints
-# Replace SOURCE_MINIO_ENDPOINT, DEST_MINIO_ENDPOINT, ACCESS_KEY, and SECRET_KEY with your values
-mc alias set source https://SOURCE_MINIO_ENDPOINT/ ACCESS_KEY SECRET_KEY
-mc alias set dest https://DEST_MINIO_ENDPOINT/ ACCESS_KEY SECRET_KEY
-
-# Example:
-# mc alias set dest https://minio.\<mydomain\>/ myuser mypassword
+# Set up source and destination S3 endpoints
+# Replace SOURCE_S3_ENDPOINT, DEST_S3_ENDPOINT, ACCESS_KEY, and SECRET_KEY with your values
+mc alias set source https://SOURCE_S3_ENDPOINT/ ACCESS_KEY SECRET_KEY
+mc alias set dest https://DEST_S3_ENDPOINT/ ACCESS_KEY SECRET_KEY
+
+# Example for SeaweedFS:
+# mc alias set source https://seaweed.\<mydomain\>/ myuser mypassword
+# mc alias set dest https://seaweed-backup.\<mydomain\>/ myuser mypassword
 ```
 
 ### Step 2: Enable Versioning (Required for Replication)
@@ -58,9 +59,9 @@ mc replicate add dest/DEST_BUCKET_NAME/ \
    --remote-bucket 'https://ACCESS_KEY:SECRET_KEY@SOURCE_MINIO_ENDPOINT/SOURCE_BUCKET_NAME' \
    --replicate "delete,delete-marker,existing-objects"
 
-# Example:
+# Example for SeaweedFS:
 # mc replicate add source/my-source-bucket/ \
-#    --remote-bucket 'https://myuser:mypassword@minio.example.com/my-dest-bucket' \
+#    --remote-bucket 'https://myuser:mypassword@seaweed.example.com/my-dest-bucket' \
 #    --replicate "delete,delete-marker,existing-objects"
 ```
 
@@ -74,95 +75,95 @@ mc replicate resync start dest/DEST_BUCKET_NAME/ --remote-bucket $(mc replicate
 # mc replicate resync start dest/my-dest-bucket/ --remote-bucket $(mc replicate status dest/my-dest-bucket/ --json | jq -r '.remoteTargets[].arn')
 ```
 
-## 2. MinIO One-Time Backup to Filesystem (e.g. local or NFS)
+## 2. SeaweedFS One-Time Backup to Filesystem (e.g. local or NFS)
 
 **Note:** This example uses NFS. If using local filesystem, skip the mount steps.
 
 ### Create mount point
 ```bash
 # Creates the directory where NFS will be mounted
-sudo mkdir -p /mnt/minio-backup
+sudo mkdir -p /mnt/s3-backup
 ```
 
 ### Mount NFS share
 ```bash
 # Mounts the NFS share to the local directory
-# Replace NFS_SERVER and /path/to/minio/backup with your NFS server details
-sudo mount -t nfs NFS_SERVER:/path/to/minio/backup /mnt/minio-backup
+# Replace NFS_SERVER and /path/to/s3/backup with your NFS server details
+sudo mount -t nfs NFS_SERVER:/path/to/s3/backup /mnt/s3-backup
 ```
 
 ### Create Backup
 ```bash
 # Create backup directory with timestamp
 # This creates a unique directory for this backup using current date and time
-mkdir -p /mnt/minio-backup/backup-$(date +%Y-%m-%d_%H-%M)
+mkdir -p /mnt/s3-backup/backup-$(date +%Y-%m-%d_%H-%M)
 
-# Set up source MinIO endpoints
-# Use "CONSOLE_ACCESS_KEY" from the default-user secret (has required permissions)
-mc alias set source https://SOURCE_MINIO_ENDPOINT/ ACCESS_KEY SECRET_KEY
+# Set up source SeaweedFS S3 endpoint
+# Get credentials from the seaweedfs-s3-config secret (ApiUser identity)
+mc alias set source https://SOURCE_S3_ENDPOINT/ ACCESS_KEY SECRET_KEY
 
-# Example:
-# mc alias set source https://minio.\<mydomain\>/ myuser mypassword
+# Example for in-cluster SeaweedFS:
+# mc alias set source https://seaweed.\<mydomain\>/ myuser mypassword
 
 # Mirror bucket contents to NFS
-# This copies all files from the MinIO bucket to the NFS backup location
-mc mirror source/BUCKET_NAME/ /mnt/minio-backup/backup-$(date +%Y-%m-%d_%H-%M)/BUCKET_NAME/ --overwrite
+# This copies all files from the S3 bucket to the NFS backup location
+mc mirror source/BUCKET_NAME/ /mnt/s3-backup/backup-$(date +%Y-%m-%d_%H-%M)/BUCKET_NAME/ --overwrite
 
 # Example:
-# mc mirror source/default-bucket/ /mnt/minio-backup/backup-$(date +%Y-%m-%d_%H-%M)/default-bucket/ --overwrite 
+# mc mirror source/default-bucket/ /mnt/s3-backup/backup-$(date +%Y-%m-%d_%H-%M)/default-bucket/ --overwrite 
 
 # Unmount when done
 # Safely disconnects the NFS share
-sudo umount /mnt/minio-backup
+sudo umount /mnt/s3-backup
 ```
 
 ### Verify Backup
 ```bash
 # Re-mount and check
 # Reconnects to the NFS share to verify the backup
-sudo mount -t nfs NFS_SERVER:/path/to/minio/backup /mnt/minio-backup
-ls -la /mnt/minio-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/
+sudo mount -t nfs NFS_SERVER:/path/to/s3/backup /mnt/s3-backup
+ls -la /mnt/s3-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/
 
 # Compare file counts
 # Verify the backup by comparing the number of files
 # Check original bucket
 mc ls --recursive source/BUCKET_NAME/ | wc -l
 
 # Check backup
-find /mnt/minio-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/ -type f | wc -l
+find /mnt/s3-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/ -type f | wc -l
 ```
 
 ### Restore from NFS Backup
 ```bash
 # Mount NFS share
 # Connects to the NFS share containing backups
-sudo mount -t nfs NFS_SERVER:/path/to/minio/backup /mnt/minio-backup
+sudo mount -t nfs NFS_SERVER:/path/to/s3/backup /mnt/s3-backup
 
 # List available backups
 # Shows all backup directories to help you choose which one to restore
-ls -la /mnt/minio-backup/
+ls -la /mnt/s3-backup/
 
 # Restore from specific backup date
-# Mirrors files from the backup back to the MinIO bucket
+# Mirrors files from the backup back to the S3 bucket
 # Replace YYYY-MM-DD_HH-MM with your actual backup directory name
-mc mirror /mnt/minio-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/ source/BUCKET_NAME/ --overwrite
+mc mirror /mnt/s3-backup/backup-YYYY-MM-DD_HH-MM/BUCKET_NAME/ source/BUCKET_NAME/ --overwrite
 
 # Example:
-# mc mirror /mnt/minio-backup/backup-2024-11-24_14-30/default-bucket/ source/default-bucket/ --overwrite
+# mc mirror /mnt/s3-backup/backup-2024-11-24_14-30/default-bucket/ source/default-bucket/ --overwrite
 
 # Verify restoration
 # Lists files in the bucket to confirm restoration
 mc ls source/BUCKET_NAME/
 
 # Unmount when done
 # Safely disconnects the NFS share
-sudo umount /mnt/minio-backup
+sudo umount /mnt/s3-backup
 ```
 
 ## What These Commands Do
 
 **Two-Way Replication:**
-1. Sets up aliases for source and destination MinIO endpoints
+1. Sets up aliases for source and destination S3-compatible endpoints
 2. Enables versioning on both buckets (required for replication)
 3. Creates bidirectional replication rules
 4. Provides a command to resync from backup when source fails
@@ -172,8 +173,10 @@ sudo umount /mnt/minio-backup
 2. Creates a timestamped backup directory
 3. Uses `mc mirror` to copy all bucket contents to the backup location
 4. Provides verification commands to compare file counts
-5. Provides restore commands to mirror files back to MinIO
+5. Provides restore commands to mirror files back to SeaweedFS
 
 **Benefits:**
 - **Replication:** Continuous, automatic backup with minimal data loss
 - **One-Time Backup:** Point-in-time snapshots stored on separate storage
+
+**Note:** SeaweedFS provides S3-compatible storage, so the MinIO client (`mc`) works seamlessly for backup and restore operations.