Skip to content

README.md

Latest commit

 

History

History
272 lines (212 loc) · 11.5 KB

README.md

File metadata and controls

272 lines (212 loc) · 11.5 KB
Raw

VariObjectStorage (VOS)

Lint Tests Coverage

A cloud-agnostic object storage gateway that provides a unified S3-compatible API across Azure Blob Storage, Google Cloud Storage, Amazon S3, and local disk backends.

Philosophy

Cloud agnosticism over feature completeness.

VOS prioritizes consistent behavior across all backends over implementing features that only work on specific platforms. This means:

  • Every S3 API operation behaves identically regardless of the underlying storage backend
  • Applications can switch between cloud providers without code changes or behavioral differences
  • Features are only implemented when they can work consistently across all supported backends

This approach ensures that your application remains truly portable and avoids vendor lock-in.

Overview

VOS enables applications to interact with multiple cloud storage providers through a single, unified S3-compatible API. This allows for:

  • True Cloud Agnosticism: Consistent behavior across all backends—no surprises when switching providers
  • S3 Compatibility: Use existing S3 SDKs and tools with any supported backend
  • Vendor Flexibility: Switch between Amazon S3, Azure Blob Storage, and Google Cloud Storage without changing application code
  • Simplified Migration: Move between cloud providers with confidence that your application will work the same way

Architecture

┌──────────────────────────────────────────────────────────┐
│                    S3 API Frontend                       │
│                  (S3-compatible REST API)                │
└────────────────────────────┬─────────────────────────────┘
                             │
                  ┌──────────┴──────────┐
                  │ Storage Abstraction │
                  │        Layer        │
                  └──────────┬──────────┘
                             │
      ┌──────────────┬───────┴──────┬──────────────┐
      │              │              │              │
      ▼              ▼              ▼              ▼
┌───────────┐ ┌─────────────┐ ┌───────────┐ ┌────────────┐
│  Amazon   │ │  Azure Blob │ │  Google   │ │  Local     │
│  S3       │ │  Storage    │ │  Cloud    │ │  Disk      │
│  Backend  │ │  Backend    │ │  Storage  │ │  Backend   │
└───────────┘ └─────────────┘ └───────────┘ └────────────┘

S3 API Feature Support

For detailed implementation status of all S3 API features and backend support, see STATUS.md.

Note: Features are only marked as supported when they work consistently across all backends. Backend-specific functionality is intentionally not exposed to maintain cloud agnosticism.

Configuration

VOS uses credentials files to securely configure authentication and backend access. See config.example.yaml for a complete example.

Environment Variables

All configuration options can be set via environment variables instead of a config file. Environment variables use the VOS_ prefix and replace dots with underscores:

Config Key Environment Variable
server.host VOS_SERVER_HOST
server.port VOS_SERVER_PORT
server.shutdown_timeout VOS_SERVER_SHUTDOWN_TIMEOUT
logging.level VOS_LOGGING_LEVEL
logging.format VOS_LOGGING_FORMAT
auth.enabled VOS_AUTH_ENABLED
auth.allow_anonymous VOS_AUTH_ALLOW_ANONYMOUS
auth.region VOS_AUTH_REGION
auth.credentials_file VOS_AUTH_CREDENTIALS_FILE
backends.default VOS_BACKENDS_DEFAULT
backends.aws-s3.region VOS_BACKENDS_AWS_S3_REGION
backends.aws-s3.credentials_file VOS_BACKENDS_AWS_S3_CREDENTIALS_FILE
backends.aws-s3.endpoint VOS_BACKENDS_AWS_S3_ENDPOINT
backends.azblob.credentials_file VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE
backends.azblob.endpoint VOS_BACKENDS_AZBLOB_ENDPOINT
backends.gcs.project_id VOS_BACKENDS_GCS_PROJECT_ID
backends.gcs.credentials_file VOS_BACKENDS_GCS_CREDENTIALS_FILE
backends.gcs.endpoint VOS_BACKENDS_GCS_ENDPOINT
backends.disk.root_dir VOS_BACKENDS_DISK_ROOT_DIR

Example using environment variables:

export VOS_SERVER_PORT=9000
export VOS_BACKENDS_DEFAULT=azblob
export VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE=/secrets/azure-credentials.json
export VOS_AUTH_ENABLED=true
export VOS_AUTH_CREDENTIALS_FILE=/secrets/auth-credentials.json

./vos serve

Environment variables take precedence over config file values, allowing you to override specific settings without modifying the config file.

Credentials Files

All sensitive credentials are stored in separate JSON files:

Auth credentials (S3 API access):

{
  "credentials": [
    {"access_key_id": "your-access-key", "secret_access_key": "your-secret-key"}
  ]
}

AWS S3 backend:

{"access_key_id": "...", "secret_access_key": "..."}

Azure Blob backend:

{"account_name": "...", "account_key": "..."}

GCS backend: Uses the standard GCP service account JSON key file.

Disk backend: No credentials required. Configure the root directory for storage:

backends:
  default: disk
  disk:
    root_dir: /var/lib/variobjectstorage

Graceful Shutdown

The server supports graceful shutdown, waiting for active uploads and downloads to complete:

server:
  shutdown_timeout: 60  # seconds to wait for active requests

Project Structure

VariObjectStorage/
├── cmd/
│   └── vos/
│       └── main.go                 # Application entry point
├── internal/
│   ├── api/
│   │   ├── routes.go               # HTTP routing setup
│   │   └── handler/
│   │       ├── handler.go          # Base handler with shared utilities
│   │       ├── bucket.go           # Bucket operation handlers
│   │       ├── object.go           # Object operation handlers
│   │       ├── multipart.go        # Multipart upload handlers
│   │       └── types.go            # S3 XML request/response types
│   ├── auth/
│   │   ├── middleware.go           # Authentication middleware
│   │   ├── middleware_test.go      # Authentication middleware tests
│   │   ├── signature.go            # AWS Signature V4 implementation
│   │   └── signature_test.go       # Signature verification tests
│   ├── backend/
│   │   ├── backend.go              # Backend factory
│   │   ├── types/
│   │   │   └── types.go            # Backend interface definition
│   │   ├── awss3/                  # Amazon S3 backend
│   │   ├── azblob/                 # Azure Blob Storage backend
│   │   ├── gcs/                    # Google Cloud Storage backend
│   │   └── disk/                   # Local disk storage backend
│   ├── cmd/
│   │   ├── root.go                 # CLI root command
│   │   └── serve.go                # Server command
│   ├── config/
│   │   ├── config.go               # Configuration and credentials loading
│   │   └── config_test.go          # Configuration tests
│   └── server/
│       └── server.go               # HTTP server with graceful shutdown
├── tests/
│   └── integration/
│       ├── main_test.go            # Test setup with testcontainers
│       ├── bucket_test.go          # Bucket operation tests
│       ├── object_test.go          # Object operation tests
│       └── multipart_test.go       # Multipart upload tests
├── dev/
│   ├── docker-compose.yaml         # Development environment
│   └── credentials/                # Development credentials files
├── config.example.yaml             # Example configuration
├── Dockerfile                      # Container build
├── Makefile                        # Build automation
└── STATUS.md                       # Implementation status tracking

Getting Started

Running with Docker

The easiest way to run VOS is using the official Docker image from GitHub Container Registry:

# Pull the latest image
docker pull ghcr.io/variamity/variobjectstorage:latest

# Run with a config file
docker run -p 8080:8080 \
  -v /path/to/config.yaml:/app/config.yaml \
  -v /path/to/credentials:/app/credentials \
  ghcr.io/variamity/variobjectstorage:latest

# Or run with environment variables
docker run -p 8080:8080 \
  -e VOS_SERVER_PORT=8080 \
  -e VOS_BACKENDS_DEFAULT=azblob \
  -e VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE=/app/credentials/azure-credentials.json \
  -e VOS_AUTH_ENABLED=true \
  -e VOS_AUTH_CREDENTIALS_FILE=/app/credentials/auth-credentials.json \
  -v /path/to/credentials:/app/credentials \
  ghcr.io/variamity/variobjectstorage:latest

Available tags:

  • latest - Latest stable release
  • X - Specific major version (e.g., 0)
  • X.Y - Specific minor version (e.g., 0.1)
  • X.Y.Z - Specific patch version (e.g., 0.1.0)

Running Tests

The integration tests run against all storage backends by default. Each test is executed as a subtest for each backend, making it easy to identify backend-specific issues:

# Run all tests against all backends (azblob, aws-s3, gcs, disk)
make test

# Run tests with a specific backend only
make test-azblob    # Azure Blob Storage backend only
make test-aws-s3    # AWS S3 backend only  
make test-gcs       # Google Cloud Storage backend only
make test-disk      # Local disk backend only

# Or using go test directly:
go test -v ./...                    # All backends
BACKEND=azblob go test -v ./...     # Specific backend
BACKEND=aws-s3,gcs go test -v ./... # Multiple backends

The tests use testcontainers to automatically spin up the appropriate emulators:

  • azblob: Azurite (Azure Storage emulator)
  • aws-s3: LocalStack (AWS emulator)
  • gcs: fake-gcs-server (Google Cloud Storage emulator)
  • disk: Uses a temporary directory (no container required)

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please ensure that any new features or changes:

  1. Work consistently across all supported backends
  2. Include integration tests that run against all backends
  3. Follow the cloud-agnostic philosophy of the project

If a feature cannot be implemented consistently across all backends, it should not be added.