Job Queue System (Async Processing Service)

A production-ready asynchronous job processing service built with Go and Redis. Designed for high-performance background task execution with reliability features like exponential backoff retries and dead letter queues.

🚀 Features

• Asynchronous Processing: Non-blocking job enqueuing via REST API using Gin Gonic.
• Robust Redis Backend:
  • Queueing: Uses Redis Lists (LPUSH/BRPOP) for primary job distribution.
  • Scheduling: Uses Redis Sorted Sets (ZSET) for delayed retries with millisecond precision.
• Reliability & Fault Tolerance:
  • Exponential Backoff: Automatic retry strategy (2^retries * 5s) to prevent system overload during transient failures.
  • Dead Letter Queue (DLQ): Permanently failed jobs are archived for manual inspection and recovery.
  • DLQ Requeue: Administrative endpoint to move jobs back to the main queue after troubleshooting.
• Observability:
  • Structured Logging: JSON-formatted logs using uber-go/zap, ready for ELK/Grafana indexing.
  • Job Lifecycle Tracking: Real-time status Monitoring (Pending, InProcess, Completed, Failed, Retrying).
• Security: Mandatory X-API-KEY authentication for all endpoints.
• Production Ready:
  • Graceful shutdown logic to prevent job loss during service restarts.
  • Environment-based configuration management.
  • Fully Dockerized with optimized multi-stage builds.

🏗️ Architecture

flowchart TD
    Client[Client App] -->|POST /job| API[API Service]
    API -->|Auth Middleware| API
    API -->|Set Status| Store[(Redis Status Store)]
    API -->|LPUSH| RedisQueue[(Redis Main Queue)]
    
    Worker[Worker Service] -->|BRPOP| RedisQueue
    Worker -->|Update Status| Store
    Worker -->|Execute| Handler[Job Handler]
    
    Handler -->|Success| Complete[Mark Completed]
    Handler -->|Fail| RetryCheck{Max Retries?}
    
    RetryCheck -->|No| Backoff[Calculate Backoff]
    Backoff -->|ZADD| RedisSched[(Redis Scheduled Set)]
    
    RetryCheck -->|Yes| DLQ[Move to DLQ]
    
    Scheduler[Retry Scheduler] -->|Poll ZRANGE| RedisSched
    RedisSched -->|Expire| RedisQueue

Loading

🛠️ Project Structure

.
├── cmd/
│   ├── api/          # API Service entry point & handlers
│   └── worker/       # Worker Service loop & scheduler
├── internal/
│   ├── config/       # Configuration management (Env vars)
│   ├── job/          # Domain models, handlers & status store
│   ├── logger/       # Structured logging setup (Zap)
│   └── queue/        # Redis interaction layer & interfaces
├── Dockerfile        # Multi-stage Docker build
└── docker-compose.yml # Service orchestration

⚙️ Configuration

The system is configured via environment variables.

Variable	Description	Default
REDIS_ADDR	Redis connection address	localhost:6379
API_PORT	Port for the API service	8081
API_KEY	Secret key for X-API-KEY header	secret-key
MAX_RETRIES	Max retry attempts per job	3
QUEUE_NAME	Primary Redis list name	jobs
DLQ_NAME	Dead letter queue list name	jobs_dlq
APP_ENV	Environment (development or production)	production

🚀 Getting Started

Prerequisites

• Docker & Docker Compose

Run the System

docker-compose up --build

• API: http://localhost:8081
• Redis: localhost:6379

🔌 API Reference

Auth Header: All requests must include X-API-KEY: <your-secret-key>.

1. Enqueue Job

POST /job

{
  "type": "email",
  "payload": "user@example.com"
}

2. Check Status

GET /job/:id

3. Requeue from DLQ

POST /job/requeue-dlq
Moves all jobs from jobs_dlq back to the main queue.

🧪 Testing

Run unit and mock-based tests:

go test ./...

📈 Scaling

The system is designed to be horizontally scalable.

docker-compose up --scale worker=5