A cloud-agnostic object storage gateway that provides a unified S3-compatible API across Azure Blob Storage, Google Cloud Storage, Amazon S3, and local disk backends.
Cloud agnosticism over feature completeness.
VOS prioritizes consistent behavior across all backends over implementing features that only work on specific platforms. This means:
- Every S3 API operation behaves identically regardless of the underlying storage backend
- Applications can switch between cloud providers without code changes or behavioral differences
- Features are only implemented when they can work consistently across all supported backends
This approach ensures that your application remains truly portable and avoids vendor lock-in.
VOS enables applications to interact with multiple cloud storage providers through a single, unified S3-compatible API. This allows for:
- True Cloud Agnosticism: Consistent behavior across all backends—no surprises when switching providers
- S3 Compatibility: Use existing S3 SDKs and tools with any supported backend
- Vendor Flexibility: Switch between Amazon S3, Azure Blob Storage, and Google Cloud Storage without changing application code
- Simplified Migration: Move between cloud providers with confidence that your application will work the same way
┌──────────────────────────────────────────────────────────┐
│ S3 API Frontend │
│ (S3-compatible REST API) │
└────────────────────────────┬─────────────────────────────┘
│
┌──────────┴──────────┐
│ Storage Abstraction │
│ Layer │
└──────────┬──────────┘
│
┌──────────────┬───────┴──────┬──────────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌───────────┐ ┌─────────────┐ ┌───────────┐ ┌────────────┐
│ Amazon │ │ Azure Blob │ │ Google │ │ Local │
│ S3 │ │ Storage │ │ Cloud │ │ Disk │
│ Backend │ │ Backend │ │ Storage │ │ Backend │
└───────────┘ └─────────────┘ └───────────┘ └────────────┘
For detailed implementation status of all S3 API features and backend support, see STATUS.md.
Note: Features are only marked as supported when they work consistently across all backends. Backend-specific functionality is intentionally not exposed to maintain cloud agnosticism.
VOS uses credentials files to securely configure authentication and backend access. See config.example.yaml for a complete example.
All configuration options can be set via environment variables instead of a config file. Environment variables use the VOS_ prefix and replace dots with underscores:
| Config Key | Environment Variable |
|---|---|
server.host |
VOS_SERVER_HOST |
server.port |
VOS_SERVER_PORT |
server.shutdown_timeout |
VOS_SERVER_SHUTDOWN_TIMEOUT |
logging.level |
VOS_LOGGING_LEVEL |
logging.format |
VOS_LOGGING_FORMAT |
auth.enabled |
VOS_AUTH_ENABLED |
auth.allow_anonymous |
VOS_AUTH_ALLOW_ANONYMOUS |
auth.region |
VOS_AUTH_REGION |
auth.credentials_file |
VOS_AUTH_CREDENTIALS_FILE |
backends.default |
VOS_BACKENDS_DEFAULT |
backends.aws-s3.region |
VOS_BACKENDS_AWS_S3_REGION |
backends.aws-s3.credentials_file |
VOS_BACKENDS_AWS_S3_CREDENTIALS_FILE |
backends.aws-s3.endpoint |
VOS_BACKENDS_AWS_S3_ENDPOINT |
backends.azblob.credentials_file |
VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE |
backends.azblob.endpoint |
VOS_BACKENDS_AZBLOB_ENDPOINT |
backends.gcs.project_id |
VOS_BACKENDS_GCS_PROJECT_ID |
backends.gcs.credentials_file |
VOS_BACKENDS_GCS_CREDENTIALS_FILE |
backends.gcs.endpoint |
VOS_BACKENDS_GCS_ENDPOINT |
backends.disk.root_dir |
VOS_BACKENDS_DISK_ROOT_DIR |
Example using environment variables:
export VOS_SERVER_PORT=9000
export VOS_BACKENDS_DEFAULT=azblob
export VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE=/secrets/azure-credentials.json
export VOS_AUTH_ENABLED=true
export VOS_AUTH_CREDENTIALS_FILE=/secrets/auth-credentials.json
./vos serveEnvironment variables take precedence over config file values, allowing you to override specific settings without modifying the config file.
All sensitive credentials are stored in separate JSON files:
Auth credentials (S3 API access):
{
"credentials": [
{"access_key_id": "your-access-key", "secret_access_key": "your-secret-key"}
]
}AWS S3 backend:
{"access_key_id": "...", "secret_access_key": "..."}Azure Blob backend:
{"account_name": "...", "account_key": "..."}GCS backend: Uses the standard GCP service account JSON key file.
Disk backend: No credentials required. Configure the root directory for storage:
backends:
default: disk
disk:
root_dir: /var/lib/variobjectstorageThe server supports graceful shutdown, waiting for active uploads and downloads to complete:
server:
shutdown_timeout: 60 # seconds to wait for active requestsVariObjectStorage/
├── cmd/
│ └── vos/
│ └── main.go # Application entry point
├── internal/
│ ├── api/
│ │ ├── routes.go # HTTP routing setup
│ │ └── handler/
│ │ ├── handler.go # Base handler with shared utilities
│ │ ├── bucket.go # Bucket operation handlers
│ │ ├── object.go # Object operation handlers
│ │ ├── multipart.go # Multipart upload handlers
│ │ └── types.go # S3 XML request/response types
│ ├── auth/
│ │ ├── middleware.go # Authentication middleware
│ │ ├── middleware_test.go # Authentication middleware tests
│ │ ├── signature.go # AWS Signature V4 implementation
│ │ └── signature_test.go # Signature verification tests
│ ├── backend/
│ │ ├── backend.go # Backend factory
│ │ ├── types/
│ │ │ └── types.go # Backend interface definition
│ │ ├── awss3/ # Amazon S3 backend
│ │ ├── azblob/ # Azure Blob Storage backend
│ │ ├── gcs/ # Google Cloud Storage backend
│ │ └── disk/ # Local disk storage backend
│ ├── cmd/
│ │ ├── root.go # CLI root command
│ │ └── serve.go # Server command
│ ├── config/
│ │ ├── config.go # Configuration and credentials loading
│ │ └── config_test.go # Configuration tests
│ └── server/
│ └── server.go # HTTP server with graceful shutdown
├── tests/
│ └── integration/
│ ├── main_test.go # Test setup with testcontainers
│ ├── bucket_test.go # Bucket operation tests
│ ├── object_test.go # Object operation tests
│ └── multipart_test.go # Multipart upload tests
├── dev/
│ ├── docker-compose.yaml # Development environment
│ └── credentials/ # Development credentials files
├── config.example.yaml # Example configuration
├── Dockerfile # Container build
├── Makefile # Build automation
└── STATUS.md # Implementation status tracking
The easiest way to run VOS is using the official Docker image from GitHub Container Registry:
# Pull the latest image
docker pull ghcr.io/variamity/variobjectstorage:latest
# Run with a config file
docker run -p 8080:8080 \
-v /path/to/config.yaml:/app/config.yaml \
-v /path/to/credentials:/app/credentials \
ghcr.io/variamity/variobjectstorage:latest
# Or run with environment variables
docker run -p 8080:8080 \
-e VOS_SERVER_PORT=8080 \
-e VOS_BACKENDS_DEFAULT=azblob \
-e VOS_BACKENDS_AZBLOB_CREDENTIALS_FILE=/app/credentials/azure-credentials.json \
-e VOS_AUTH_ENABLED=true \
-e VOS_AUTH_CREDENTIALS_FILE=/app/credentials/auth-credentials.json \
-v /path/to/credentials:/app/credentials \
ghcr.io/variamity/variobjectstorage:latestAvailable tags:
latest- Latest stable releaseX- Specific major version (e.g.,0)X.Y- Specific minor version (e.g.,0.1)X.Y.Z- Specific patch version (e.g.,0.1.0)
The integration tests run against all storage backends by default. Each test is executed as a subtest for each backend, making it easy to identify backend-specific issues:
# Run all tests against all backends (azblob, aws-s3, gcs, disk)
make test
# Run tests with a specific backend only
make test-azblob # Azure Blob Storage backend only
make test-aws-s3 # AWS S3 backend only
make test-gcs # Google Cloud Storage backend only
make test-disk # Local disk backend only
# Or using go test directly:
go test -v ./... # All backends
BACKEND=azblob go test -v ./... # Specific backend
BACKEND=aws-s3,gcs go test -v ./... # Multiple backendsThe tests use testcontainers to automatically spin up the appropriate emulators:
- azblob: Azurite (Azure Storage emulator)
- aws-s3: LocalStack (AWS emulator)
- gcs: fake-gcs-server (Google Cloud Storage emulator)
- disk: Uses a temporary directory (no container required)
This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please ensure that any new features or changes:
- Work consistently across all supported backends
- Include integration tests that run against all backends
- Follow the cloud-agnostic philosophy of the project
If a feature cannot be implemented consistently across all backends, it should not be added.