Disk backend

Dockerfile

@@ -27,8 +27,10 @@ RUN apk add --no-cache ca-certificates tzdata
 # Copy binary from builder
 COPY --from=builder /app/vos .
 
-# Create non-root user
-RUN adduser -D -g '' appuser
+# Create non-root user and data directory
+RUN adduser -D -g '' appuser && \
+    mkdir -p /data && \
+    chown -R appuser:appuser /data
 USER appuser
 
 # Expose port

README.md

@@ -4,7 +4,7 @@
 [![Tests](https://github.com/variamity/variobjectstorage/actions/workflows/test.yaml/badge.svg)](https://github.com/variamity/variobjectstorage/actions/workflows/test.yaml)
 ![Coverage](.github/badges/coverage.svg)
 
-A cloud-agnostic object storage gateway that provides a unified S3-compatible API across Azure Blob Storage, Google Cloud Storage, and Amazon S3 backends.
+A cloud-agnostic object storage gateway that provides a unified S3-compatible API across Azure Blob Storage, Google Cloud Storage, Amazon S3, and local disk backends.
 
 ## Philosophy
 
@@ -33,21 +33,21 @@ VOS enables applications to interact with multiple cloud storage providers throu
 ┌──────────────────────────────────────────────────────────┐
 │                    S3 API Frontend                       │
 │                  (S3-compatible REST API)                │
-└─────────────────────────────┬────────────────────────────┘
-                              │
-                  ┌───────────┴───────────┐
-                  │   Storage Abstraction │
-                  │         Layer         │
-                  └───────────┬───────────┘
-                              │
-      ┌──────────────┬────────┴─────┬──────────────┐
+└────────────────────────────┬─────────────────────────────┘
+                             │
+                  ┌──────────┴──────────┐
+                  │ Storage Abstraction │
+                  │        Layer        │
+                  └──────────┬──────────┘
+                             │
+      ┌──────────────┬───────┴──────┬──────────────┐
       │              │              │              │
       ▼              ▼              ▼              ▼
-┌───────────┐ ┌─────────────┐ ┌───────────┐ ┌─────────────┐
-│  Amazon   │ │  Azure Blob │ │  Google   │ │   (Future)  │
-│  S3       │ │  Storage    │ │  Cloud    │ │   Backends  │
-│  Backend  │ │  Backend    │ │  Storage  │ │             │
-└───────────┘ └─────────────┘ └───────────┘ └─────────────┘
+┌───────────┐ ┌─────────────┐ ┌───────────┐ ┌────────────┐
+│  Amazon   │ │  Azure Blob │ │  Google   │ │  Local     │
+│  S3       │ │  Storage    │ │  Cloud    │ │  Disk      │
+│  Backend  │ │  Backend    │ │  Storage  │ │  Backend   │
+└───────────┘ └─────────────┘ └───────────┘ └────────────┘
 ```
 
 ## S3 API Feature Support
@@ -65,7 +65,7 @@ VOS uses credentials files to securely configure authentication and backend acce
 All configuration options can be set via environment variables instead of a config file. Environment variables use the `VOS_` prefix and replace dots with underscores:
 
 | Config Key | Environment Variable |
-|------------|---------------------|
+| ---------- | -------------------- |
 | `server.host` | `VOS_SERVER_HOST` |
 | `server.port` | `VOS_SERVER_PORT` |
 | `server.shutdown_timeout` | `VOS_SERVER_SHUTDOWN_TIMEOUT` |
@@ -84,6 +84,7 @@ All configuration options can be set via environment variables instead of a conf
 | `backends.gcs.project_id` | `VOS_BACKENDS_GCS_PROJECT_ID` |
 | `backends.gcs.credentials_file` | `VOS_BACKENDS_GCS_CREDENTIALS_FILE` |
 | `backends.gcs.endpoint` | `VOS_BACKENDS_GCS_ENDPOINT` |
+| `backends.disk.root_dir` | `VOS_BACKENDS_DISK_ROOT_DIR` |
 
 Example using environment variables:
 
@@ -127,6 +128,15 @@ All sensitive credentials are stored in separate JSON files:
 
 **GCS backend**: Uses the standard GCP service account JSON key file.
 
+**Disk backend**: No credentials required. Configure the root directory for storage:
+
+```yaml
+backends:
+  default: disk
+  disk:
+    root_dir: /var/lib/variobjectstorage
+```
+
 ### Graceful Shutdown
 
 The server supports graceful shutdown, waiting for active uploads and downloads to complete:
@@ -163,7 +173,8 @@ VariObjectStorage/
 │   │   │   └── types.go            # Backend interface definition
 │   │   ├── awss3/                  # Amazon S3 backend
 │   │   ├── azblob/                 # Azure Blob Storage backend
-│   │   └── gcs/                    # Google Cloud Storage backend
+│   │   ├── gcs/                    # Google Cloud Storage backend
+│   │   └── disk/                   # Local disk storage backend
 │   ├── cmd/
 │   │   ├── root.go                 # CLI root command
 │   │   └── serve.go                # Server command
@@ -226,13 +237,14 @@ Available tags:
 The integration tests run against all storage backends by default. Each test is executed as a subtest for each backend, making it easy to identify backend-specific issues:
 
 ```bash
-# Run all tests against all backends (azblob, aws-s3, gcs)
+# Run all tests against all backends (azblob, aws-s3, gcs, disk)
 make test
 
 # Run tests with a specific backend only
 make test-azblob    # Azure Blob Storage backend only
 make test-aws-s3    # AWS S3 backend only  
 make test-gcs       # Google Cloud Storage backend only
+make test-disk      # Local disk backend only
 
 # Or using go test directly:
 go test -v ./...                    # All backends
@@ -245,6 +257,7 @@ The tests use [testcontainers](https://testcontainers.com/) to automatically spi
 - **azblob**: Azurite (Azure Storage emulator)
 - **aws-s3**: LocalStack (AWS emulator)
 - **gcs**: fake-gcs-server (Google Cloud Storage emulator)
+- **disk**: Uses a temporary directory (no container required)
 
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
 

STATUS.md

@@ -138,6 +138,7 @@ This document tracks the implementation progress of S3 API features and backend
 | Amazon S3 (aws-s3) | ✅ Complete | All core operations implemented |
 | Azure Blob Storage (azblob) | ✅ Complete | All core operations implemented |
 | Google Cloud Storage (gcs) | ✅ Complete | All core operations implemented |
+| Local Disk (disk) | ✅ Complete | All core operations implemented; stores data in local filesystem |
 
 ---
 

config.example.yaml

@@ -20,7 +20,7 @@ auth:
   credentials_file: /path/to/auth-credentials.json
 
 backends:
-  # Backend to use: aws-s3, azblob, or gcs
+  # Backend to use: aws-s3, azblob, gcs, or disk
   default: aws-s3
 
   # Amazon S3 Backend
@@ -45,3 +45,8 @@ backends:
     project_id: YOUR_PROJECT_ID
     # Service account credentials file (standard GCP JSON key file)
     credentials_file: /path/to/gcp-credentials.json
+
+  # Local Disk Backend
+  disk:
+    # Root directory for storing buckets and objects
+    root_dir: /var/lib/variobjectstorage

dev/docker-compose.yaml

@@ -2,7 +2,7 @@ services:
   # VariObjectStorage with Azure Blob Storage backend
   vos-azblob:
     build:
-      context: ../..
+      context: ..
       dockerfile: Dockerfile
     ports:
       - "127.0.0.1:8081:8080"
@@ -29,7 +29,7 @@ services:
   # VariObjectStorage with AWS S3 backend (LocalStack)
   vos-aws:
     build:
-      context: ../..
+      context: ..
       dockerfile: Dockerfile
     ports:
       - "127.0.0.1:8082:8080"
@@ -57,7 +57,7 @@ services:
   # VariObjectStorage with GCS backend (fake-gcs-server)
   vos-gcs:
     build:
-      context: ../..
+      context: ..
       dockerfile: Dockerfile
     ports:
       - "127.0.0.1:8083:8080"
@@ -81,6 +81,30 @@ services:
     networks:
       - storage-network
 
+  # VariObjectStorage with Disk backend (local filesystem)
+  vos-disk:
+    build:
+      context: ..
+      dockerfile: Dockerfile
+    ports:
+      - "127.0.0.1:8084:8080"
+    environment:
+      VOS_SERVER_HOST: "0.0.0.0"
+      VOS_SERVER_PORT: "8080"
+      VOS_LOGGING_LEVEL: "debug"
+      VOS_LOGGING_FORMAT: "console"
+      VOS_AUTH_ENABLED: "true"
+      VOS_AUTH_ALLOW_ANONYMOUS: "false"
+      VOS_AUTH_REGION: "us-east-1"
+      VOS_AUTH_CREDENTIALS_FILE: "/app/credentials/auth-credentials.json"
+      VOS_BACKENDS_DEFAULT: "disk"
+      VOS_BACKENDS_DISK_ROOT_DIR: "/data"
+    volumes:
+      - ./credentials:/app/credentials:ro
+      - disk-data:/data
+    networks:
+      - storage-network
+
   # Azure Blob Storage emulator
   azurite:
     image: mcr.microsoft.com/azure-storage/azurite:latest
@@ -143,3 +167,4 @@ volumes:
   azurite-data:
   localstack-data:
   gcs-data:
+  disk-data:

go.mod

@@ -3,17 +3,23 @@ module github.com/variamity/variobjectstorage
 go 1.25
 
 require (
+	cloud.google.com/go/storage v1.58.0
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.19.1
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3
 	github.com/aws/aws-sdk-go-v2 v1.41.0
 	github.com/aws/aws-sdk-go-v2/config v1.32.6
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.6
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
+	github.com/aws/smithy-go v1.24.0
+	github.com/gofrs/flock v0.13.0
+	github.com/google/uuid v1.6.0
 	github.com/gorilla/mux v1.8.1
 	github.com/spf13/cobra v1.8.0
 	github.com/spf13/viper v1.18.2
 	github.com/testcontainers/testcontainers-go v0.40.0
 	go.uber.org/zap v1.26.0
+	google.golang.org/api v0.256.0
 )
 
 require (
@@ -24,7 +30,6 @@ require (
 	cloud.google.com/go/compute/metadata v0.9.0 // indirect
 	cloud.google.com/go/iam v1.5.3 // indirect
 	cloud.google.com/go/monitoring v1.24.2 // indirect
-	cloud.google.com/go/storage v1.58.0 // indirect
 	dario.cat/mergo v1.0.2 // indirect
 	github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
 	github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
@@ -34,7 +39,6 @@ require (
 	github.com/Microsoft/go-winio v0.6.2 // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
 	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
@@ -47,7 +51,6 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
-	github.com/aws/smithy-go v1.24.0 // indirect
 	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/cncf/xds/go v0.0.0-20250501225837-2ac532fd4443 // indirect
@@ -71,7 +74,6 @@ require (
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-ole/go-ole v1.2.6 // indirect
 	github.com/google/s2a-go v0.1.9 // indirect
-	github.com/google/uuid v1.6.0 // indirect
 	github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
 	github.com/googleapis/gax-go/v2 v2.15.0 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
@@ -121,7 +123,6 @@ require (
 	go.opentelemetry.io/otel/sdk/metric v1.39.0 // indirect
 	go.opentelemetry.io/otel/trace v1.39.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.9.0 // indirect
-	go.uber.org/goleak v1.3.0 // indirect
 	go.uber.org/multierr v1.10.0 // indirect
 	golang.org/x/crypto v0.43.0 // indirect
 	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
@@ -131,7 +132,6 @@ require (
 	golang.org/x/sys v0.39.0 // indirect
 	golang.org/x/text v0.30.0 // indirect
 	golang.org/x/time v0.14.0 // indirect
-	google.golang.org/api v0.256.0 // indirect
 	google.golang.org/genproto v0.0.0-20250922171735-9219d122eba9 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20251111163417-95abcf5c77ba // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20251111163417-95abcf5c77ba // indirect