Shieldly DLP Backend

Shieldly is a self-hosted, multi-tenant DLP-style backend built in Go. It ingests file metadata, queues scan jobs, evaluates policies, and writes immutable audit logs. The design keeps Postgres as the source of truth and uses a bounded worker pool for predictable concurrency.

Architecture

Single Go service (HTTP API + worker loop)
Postgres 18 as the durable job and data store
Bounded worker pool with FOR UPDATE SKIP LOCKED job claiming
Idempotent ingestion via (tenant_id, content_hash)
Immutable audit log for every decision

Requirements

Go 1.25
Docker + docker-compose
Postgres 18 (via docker-compose)
sqlc for query generation

Running locally

Start Postgres:

docker compose up -d

Apply migrations:

docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/001_create_tables.sql
docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/002_add_api_keys.sql
docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/003_add_api_key_roles.sql
docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/004_policy_versions.sql
docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/005_api_key_expiry_rotation.sql
docker compose exec -T postgres psql -U dlp -d dlp -v ON_ERROR_STOP=1 -f - < db/migrations/006_add_api_key_revoke_after.sql

Generate sqlc code:

sqlc generate

Run the API:

go run ./cmd/dlpapi

Authentication (API keys + roles)

Requests require X-API-Key. The API key is hashed with SHA-256 and matched against api_keys.key_hash. X-Tenant-ID is optional; when provided it must match the tenant of the API key. API keys can expire and be revoked. API keys have roles:

admin: full access
reader: read-only (policy writes are blocked)

Example seed (replace UUIDs):

INSERT INTO tenants (id, name) VALUES ('00000000-0000-0000-0000-000000000001', 'Acme');
INSERT INTO api_keys (tenant_id, key_hash, name, role)
VALUES (
  '00000000-0000-0000-0000-000000000001',
  encode(digest('dev-key-1', 'sha256'), 'hex'),
  'local-dev',
  'admin'
);

API key CLI

shieldlyctl is a small CLI for admin API key operations.

Examples:

go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd list
go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd create -name "ops" -role admin
go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd rotate -id <key-id>
go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd revoke -id <key-id>
go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd tenant-get
go run ./cmd/shieldlyctl -api-key dev-key-1 -cmd tenant-set -max-files 1000 -max-bytes 10485760

Endpoints

All endpoints require X-API-Key (and optionally X-Tenant-ID).

Files

POST /v1/files ingest metadata
GET /v1/files/{id} fetch file
GET /v1/files?status=pending&limit=50&before=2026-01-01T00:00:00Z list files

Example ingest:

curl -X POST http://localhost:8080/v1/files \
  -H "X-API-Key: dev-key-1" \
  -H "Content-Type: application/json" \
  -d '{"content_hash":"sha256:abc","filename":"report.pdf","mime_type":"application/pdf","size_bytes":12345}'

Audit logs

Requires role: admin or reader
GET /v1/audit?event_type=scan_decided&entity_type=job&limit=100
GET /v1/audit?event_type=api_key_created&entity_type=api_key&limit=50
GET /v1/audit/export?event_type=api_key_created&entity_type=api_key&limit=1000 (NDJSON)

Export limits: limit is capped at 2000 per request.

Export access can be restricted to admin only via AUDIT_EXPORT_ADMIN_ONLY (default true).

Export access can be restricted to admin only via AUDIT_EXPORT_ADMIN_ONLY.

Policies

GET /v1/policies
GET /v1/policies/{id}/versions
GET /v1/policies/{id}/versions/diff?from=1&to=2
POST /v1/policies
PATCH /v1/policies/{id}
DELETE /v1/policies/{id}

Policy JSON example:

{
  "name": "Block large binaries",
  "enabled": true,
  "rules_json": {
    "max_size_bytes": 5000000,
    "block_mime_types": ["application/x-msdownload"]
  }
}

Policy evaluation

Worker decisions are based on enabled policies. If any policy blocks by mime type or size, the decision is block; otherwise it is allow. Each decision is recorded in the audit log with a reason string like policy:<name>:block_mime_type.

Audit retention

Configure automated cleanup with:

AUDIT_RETENTION_DAYS (0 disables cleanup)
AUDIT_RETENTION_INTERVAL (default 24h)
AUDIT_RETENTION_DRY_RUN (true logs counts without deleting)

API keys

GET /v1/api-keys?status=active|expired|revoked
POST /v1/api-keys
DELETE /v1/api-keys/{id}
POST /v1/api-keys/{id}/rotate

Rotation grace period can be configured with API_KEY_ROTATION_GRACE (e.g. 15m). When set, rotated keys remain valid until the grace window ends.

Scheduled revocations are finalized by a background job. Configure its interval with API_KEY_CLEANUP_INTERVAL (default 5m).

Dead jobs

GET /v1/jobs/dead?limit=50&before=2026-01-01T00:00:00Z
POST /v1/jobs/dead/replay

Replay body example:

{
  "job_ids": ["uuid-1", "uuid-2"]
}

Replay limits are controlled by DEAD_JOB_REPLAY_LIMIT (default 100).

Rate limiting

RATE_LIMIT_RPS (requests per second, 0 disables)
RATE_LIMIT_BURST (burst size)
RATE_LIMIT_EXPORT_RPS (export requests per second)
RATE_LIMIT_EXPORT_BURST (export burst size)
RATE_LIMIT_INGEST_RPS (ingest requests per second)
RATE_LIMIT_INGEST_BURST (ingest burst size)

Rate limit headers:

X-RateLimit-Remaining
X-RateLimit-Route
Retry-After (on 429)

Metrics

Prometheus metrics are exposed at GET /metrics (no auth).

Key metrics:

shieldly_http_requests_total
shieldly_http_request_duration_seconds
shieldly_http_errors_total
shieldly_http_inflight
shieldly_scan_jobs_total
shieldly_worker_queue_depth
shieldly_api_key_events_total
shieldly_rate_limit_total
shieldly_api_key_expiring
shieldly_tenant_requests_total
shieldly_tenant_quota_exceeded_total
shieldly_tenant_files_24h
shieldly_tenant_bytes_24h
shieldly_tenant_max_files_per_day
shieldly_tenant_max_bytes_per_day

Sample alert rules are in observability/alerts.yml.

Grafana dashboard JSON is in observability/grafana_dashboard.json.

Prometheus scrape config example is in observability/prometheus.yml.

Alertmanager routing example is in observability/alertmanager.yml.

Logging

Requests emit structured JSON logs with request_id, tenant_id, route, status, and duration_ms.

Errors include error_code and request_id in the JSON response.

Security posture

Encryption at rest is provided by the underlying Postgres storage (cloud-managed or disk encryption for local volumes).
Secrets (API keys) are stored as SHA-256 hashes; raw keys are never stored.
Audit export is scoped by tenant and capped to 2000 records per request.
Rate limiting is enforced per tenant; consider tighter limits for export endpoints.

Tracing

Tracing is optional and uses OTLP over gRPC. Enable with:

TRACING_ENABLED
TRACING_ENDPOINT
TRACING_SERVICE_NAME (default shieldly)
TRACING_INSECURE (default true)

Config validation

Configuration is validated at startup; invalid settings cause the process to exit.

API key expiry alerts

API_KEY_EXPIRY_ALERT_WINDOW (default 168h)
API_KEY_EXPIRY_ALERT_INTERVAL (default 15m)

Worker backoff

BACKOFF_BASE (default 1s)
BACKOFF_MAX (default 2m)

Tests

Integration tests use Docker (Postgres 18). Run:

go test ./...

E2E smoke test (requires jq):

API_KEY=<admin-key> make e2e

Load test (requires k6):

API_KEY=<admin-key> make loadtest

Local seed helper:

make seed

Runbooks

Operational runbooks live in docs/runbooks/.

Alert routing guidance is in docs/operations/alerts.md.

Secrets

Secrets guidance is documented in docs/security/secrets.md.

Use DATABASE_URL_FILE if your secret store mounts the database URL as a file.

Security checklist is in docs/security/checklist.md.

Config example

Sample environment file: .env.example.

Threat model

Threat model and mitigations are documented in docs/security/threat-model.md.

Security acceptance checklist is in docs/security/acceptance.md.

Deployment

Deployment guidance is documented in docs/deployment/README.md.

Staging/prod checklist is in docs/deployment/checklist.md.

Docker

Build the image:

docker build -t shieldly:local .

On-call

On-call checklist is in docs/operations/oncall.md.

SLOs and capacity targets are in docs/operations/slo.md.

Make targets

make up
make down
make migrate
make sqlc
make test
make build
make clean
make loadtest

Tenant quotas

GET /v1/tenant
PATCH /v1/tenant

Quota updates are recorded as tenant_quota_updated in the audit log.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.github/workflows		.github/workflows
cmd		cmd
db		db
docs		docs
internal		internal
observability		observability
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum
sqlc.yaml		sqlc.yaml

Zorig/Shieldly

Folders and files

Latest commit

History

Repository files navigation