Blueberry

Blueberry aims to be a light-weight server OS for aarch64-SBC's like the Raspberry Pi 4, heavily influenced by Universal Blue, in particular uCore, but built on Fedora IOT. Like uCore, it's an opinionated, "batteries included" custom image, built daily with some common tools added in.

Blueberry provides two aarch64 images with a clear layering model:

Image Family

blueberry-minimal (base container host)
        ↓
blueberry (storage primitives + observability)
        ↓
blueberry-k3s (lightweight Kubernetes)

Blueberry Minimal

The foundation image for containerized workloads on aarch64 SBCs. Based on uCore minimal, suitable for running containers on systems supported by Fedora IOT.

• Starts with a Fedora IOT image
• Adds the following:
  • bootc (new way to update container native systems)
  • cockpit (podman container and system management)
  • firewalld
  • guest VM agents (qemu-guest-agent and open-vm-tools))
  • podman-compose podman is pre-installed in Fedora IOT
  • tailscale and wireguard-tools
  • tmux
• Enables staging of automatic system updates via rpm-ostreed
• Enables password based SSH auth (required for locally running cockpit web interface)

Use blueberry-minimal when: You need a clean, minimal atomic container host without storage or monitoring tooling.

Installation (blueberry-minimal)

To rebase an existing Fedora IoT system to Blueberry Minimal:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/philbudden/blueberry-minimal:latest
systemctl reboot

Blueberry

Builds on blueberry-minimal with storage primitives and observability for edge NAS workloads and Kubernetes nodes.

Additional packages:

Storage Primitives:

• smartmontools - Disk health monitoring (SMART)
• hdparm - Low-level disk parameter management
• mdadm - Software RAID for multi-disk resilience
• cockpit-storaged - Cockpit storage management UI

Observability:

• pcp-zeroconf - Performance Co-Pilot monitoring

User Experience:

• just - Command runner for task automation
• ujust - User-friendly wrapper for just (see ujust Usage)

Design Philosophy:

• Provides primitives, not policy
• Storage services (SMB, NFS, SnapRAID, MergerFS) run in containers, not on the host
• Lightweight monitoring suitable for SBC constraints (4-8GB RAM, USB storage)
• No ZFS (inappropriate for SBC/USB storage environments)

Use blueberry when: You need storage primitives for containerized NAS workloads, Kubernetes persistent volumes, or edge storage nodes.

Installation (blueberry)

To rebase an existing Fedora IoT system to Blueberry:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/philbudden/blueberry:latest
systemctl reboot

Package Rationale

Why mdadm (Software RAID)?

While USB storage is the primary target for SBC environments, multi-USB RAID configurations are legitimate and useful for:

• Data redundancy on edge nodes (RAID1 mirroring)
• Kubernetes persistent volume backing with resilience
• Containerized NAS services requiring underlying RAID
• Lightweight alternative to ZFS (which is too heavy for SBC/USB constraints)

mdadm is the only viable RAID primitive for this environment—no datacenter assumptions, minimal overhead, suitable for USB-connected drives.

ujust Task Runner

Blueberry includes ujust, a user-friendly task runner system for common system management operations.

Usage:

# List available tasks
ujust

# Interactive task selection
ujust --choose

# Run a specific task
ujust system-info
ujust verify-storage-tools

Available tasks:

• system-info - Display system information (image, kernel, storage, memory, network)
• image-info - Show current image version and update status
• logs-this-boot - Show system logs from current boot
• logs-last-boot - Show system logs from previous boot
• verify-storage-tools - Verify all storage primitives are installed
• monitoring-status - Show PCP monitoring service status

Users can extend ujust by adding custom tasks in /usr/share/ublue-os/just/60-custom.just (create this file to add your own recipes).

Blueberry K3s

Builds on blueberry with K3s for lightweight Kubernetes workloads on SBCs.

K3s Configuration:

• Version: v1.31.4+k3s1 (pinned per image release)
• Installation: Binary distribution (single k3s binary includes server, agent, kubectl, crictl)
• Default state: Disabled (must be explicitly initialized via ujust)
• Modes: Server (control plane) or Agent (worker node)

Additional packages:

• iptables - Required for K3s network policy

Design Philosophy:

• K3s disabled by default - no automatic cluster initialization
• Explicit server/agent mode selection via ujust
• Single-node bootstrap capable (no HA assumptions)
• Version skew detection prevents starting with incompatible state
• Rollback-aware (/var state persists, version checks prevent corruption)

Use blueberry-k3s when: You need lightweight Kubernetes for edge workloads, container orchestration, or learning Kubernetes on SBC hardware.

Installation (blueberry-k3s)

To rebase an existing Fedora IoT system to Blueberry K3s:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/philbudden/blueberry-k3s:latest
systemctl reboot

K3s Bootstrap Workflow

Initialize as server (control plane):

ujust k3s-init-server

Enable kubectl without sudo:

ujust k3s-kubeconfig-user  # Copy kubeconfig to ~/.kube/config

Initialize as agent (worker node):

ujust k3s-init-agent
# You will be prompted for:
#   - Server URL (e.g., https://192.168.1.100:6443)
#   - Server token (get from server: sudo cat /var/lib/rancher/k3s/server/node-token)

Check K3s status:

ujust k3s-status       # Show service status and cluster info
ujust k3s-version      # Show binary and state versions
ujust k3s-logs         # Follow K3s logs

Bootstrap FluxCD (GitOps):

ujust flux-bootstrap-github  # Interactive FluxCD bootstrap
ujust flux-status            # Check FluxCD installation
ujust flux-version           # Show FluxCD CLI version

Get server token (for adding agents):

ujust k3s-get-token

Reset K3s (destructive):

ujust k3s-reset        # Removes all K3s state and configuration

Version Pinning & Upgrades

Version Tracking:

• Binary version: /etc/blueberry-k3s/version (immutable, part of image)
• State version: /var/lib/rancher/k3s/.version (mutable, created on init)

Upgrade Process:

1. When a new blueberry-k3s image is released with a newer K3s version
2. After rpm-ostree upgrade, K3s will detect version mismatch
3. K3s services will refuse to start (prevents state corruption)
4. Manual upgrade required (mechanism TBD)

Rollback Implications:

• rpm-ostree rollback reverts the image (including K3s binary)
• /var/lib/rancher/k3s state persists (not reverted)
• Version check prevents older binary from starting against newer state
• Resolution: Either roll forward or ujust k3s-reset (destructive)

Current Limitations:

• In-place K3s upgrades not yet implemented
• Downgrades require k3s-reset (destructive)
• Multi-version state migration not supported

Package Rationale

Why K3s over full Kubernetes?

K3s is specifically designed for resource-constrained environments:

• Single binary (< 100MB) vs full Kubernetes distribution
• Lower memory footprint (suitable for 4-8GB SBC RAM)
• Bundled components (no separate etcd, no cloud provider dependencies)
• SQLite default datastore (appropriate for single-node)
• Binary distribution (no package dependencies, deterministic versioning)

K3s is a CNCF-certified Kubernetes distribution suitable for production edge workloads.

Why binary installation over package manager?

• Deterministic versioning (exact K3s release pinned in image)
• No external repository dependencies
• Atomic updates via bootc/rpm-ostree
• Follows GitOps principles (version in source control)

Users can extend ujust by adding custom tasks in /usr/share/ublue-os/just/60-custom.just (create this file to add your own recipes).

General Architecture & Installation

• Architecture: aarch64 only (for SBCs like Raspberry Pi 4)
• Installation method: Rebase from existing Fedora IoT installation only
• Not supported: Fresh installations, ISO/disk images, x86_64 architecture

To rebase an existing Fedora IoT system to Blueberry Minimal:

rpm-ostree rebase ostree-unverified-registry:ghcr.io/philbudden/blueberry-minimal:latest
systemctl reboot

Important

Per cockpit's instructions the cockpit-ws RPM is not installed, rather it is provided as a pre-defined systemd service which runs a podman container.

Note

Key differences between Blueberry Minimal and uCore-minimal:

• Architecture: aarch64 only (uCore supports x86_64 and aarch64)
• Installation: Rebase-only (uCore supports fresh installs via ISO/disk images)
• Container tools: Given the focus on SBC hardware, a single container engine is preferred. Podman is provided out-of-the-box with Fedora IoT.
• udev rules: Not required, as only devices already supported by Fedora IoT are currently in scope.
• ZFS: Generally discouraged on SBCs due to poor performance with USB-based storage.
• NVIDIA support: While some older GPUs have been adapted for Raspberry Pi devices, this remains a rare and non-standard use case.

Repository Structure

The repository follows uCore's organizational conventions:

blueberry-minimal/                  # Minimal container host image
├── Containerfile                   # Image build definition
├── install-blueberry-minimal.sh    # Package installation script
├── cleanup.sh                      # Image cleanup script
└── system_files/                   # System configuration hierarchy
    ├── etc/                        # System configuration files
    │   └── ssh/sshd_config.d/      # SSH configuration
    └── usr/lib/systemd/system/     # Systemd unit files

blueberry/                          # Storage + observability image
├── Containerfile                   # Builds FROM blueberry-minimal
├── install-blueberry.sh            # Additional package installation
├── cleanup.sh                      # Image cleanup script
└── system_files/                   # Additional config (if needed)

This structure:

• Separates build logic from system configuration
• Mirrors the Linux filesystem hierarchy for clarity
• Enables clean multi-image support with layered variants (blueberry-minimal → blueberry → blueberry-k3s)
• Maintains compatibility with uCore patterns

Build & Release

Blueberry follows uCore's build workflow conventions:

• Workflow organization: Version-specific workflows (build-43.yml, build-44.yml) delegate to a reusable workflow (reusable-build.yml)
• Build schedule: Daily builds at 2:30 UTC (Fedora 44) and 2:35 UTC (Fedora 43)
• Image registry: GitHub Container Registry (GHCR)
• Image signing: All published images are signed with Cosign
• Build order: Images build sequentially to respect dependencies (blueberry-minimal → blueberry → blueberry-k3s)
• Tags:
  • latest - Most recent build of the default version (Fedora 44)
  • YYYYMMDD - Daily dated builds (e.g., 20260214)

Images are built only for aarch64 architecture and are intended for rebase-only installation on existing Fedora IoT systems.