Wodore Backend

wodo.re backend implementation

wodo.re ● api.wodore.com
docker images – documentation

 

Used Stack

Production

• Django with django ninja for the API and unfold admin
• PostgreSQL with PostGIS for the database
• Martin for vector tile serving
• Imagor for image serving and processing
• Zitadel for authentication and user management (optional)

Dev Tools

• uv for package management
• infisical for secrets management (optional)

Development

Initial Setup

Check Prerequisites for required tools.

When first cloning the repository:

# Install Python packages and set up virtualenv
make init
# or
uv sync
uv run invoke install
# afterwards activate the virtual environment
source .venv/bin/activate

# Initialize the persistent cache database (required for image caching)
app migrate
app createcachetable

Cache Setup (Important)

The image aggregation system uses a persistent database cache to store results from external APIs. This improves performance by 92% (from 668ms to 53ms for cached requests).

Initialize Cache Table

# Create the cache database table (run once during setup)
app createcachetable

# Verify cache is working
app shell -c "from django.core.cache import caches; print('Cache:', caches['persistent'].__class__.__name__)"

Note: This can be run multiple times safely (e.g., in Kubernetes init containers). The command is idempotent and will only create the table if it doesn't exist.

Kubernetes Pre-Init Job Example

apiVersion: batch/v1
kind: Job
metadata:
  name: cache-init
spec:
  template:
    spec:
      containers:
      - name: cache-init
        image: wodore-backend:latest
        command:
          - python
          - manage.py
          - createcachetable
        env:
          - name: DATABASE_URL
            valueFrom:
              secretKeyRef:
                name: database-secret
                key: url
      restartPolicy: OnFailure

Cache Management

Django provides built-in cache management:

# Create cache table
app createcachetable

# Clear cache (all entries)
app shell -c "from django.core.cache import caches; caches['persistent'].clear()"

# Check cache backend
app shell -c "from django.core.cache import caches; print(caches['persistent'])"

Force Cache Refresh

To bypass cache and refresh data from external APIs:

# Use update_cache parameter in API requests
curl "http://localhost:8000/v1/geo/images/hut/gelmer?update_cache=true"

For more details on caching implementation, see the internal documentation.

Setup

Activate the virtual environment and install packages:

source .venv/bin/activate

# With infisical (recommended) -> see Secrets section
(.venv) inv install --infisical

# Without infisical
(.venv) inv install

# View available commands
(.venv) inv help

# Apply changes
source deactivate; source .venv/bin/activate

NOTE: The install command creates .volumes/pgdata/ for PostgreSQL data and media/imagor_data/ for image processing.

Secrets

Secrets are managed with infisical. Install the CLI tool following the installation guide and initialize it:

infisical login
infisical init

Set up secrets using infisical (recommended):

(.venv) inv install --infisical
source .venv/bin/activate
(.venv) app <cmd> # uses infisical

Or use local env files:

# Export secrets to config/.env (update when secrets change)
infisical export --env dev --path /backend >> config/.env
ln -s config/.env .env

Or set up manually:

# Create and edit env files manually
cp config/.env.template config/.env
ln -s config/.env .env
# edit .env

TIP: Add -i/--infisical to inv commands (e.g., run, docker-compose) to use infisical directly.

Start Database and Image Service

Start PostgreSQL and Imagor services after each system restart:

# With infisical (recommended)
(.venv) inv docker-compose -c "up -d" -i

# Without infisical (requires .env file)
(.venv) inv docker-compose -c "up -d"

NOTE: PostgreSQL data is stored in .volumes/pgdata/ (development only). To reset the database:

rm -rf .volumes/pgdata/*  # Be careful!
(.venv) inv docker-compose -c "up -d"

Required PostgreSQL Extensions

The application requires the following PostgreSQL extensions for full functionality:

• postgis - Geographic objects and spatial queries (already included in PostGIS image)
• pg_trgm - Trigram similarity for fuzzy search and typo tolerance
• unaccent - Accent-insensitive text search (optional but recommended)

These extensions are installed automatically via Django migrations when you run app migrate. If you need to install them manually (e.g., on a production database):

# Development (Docker)
docker compose exec db psql -U wodore -d wodore -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
docker compose exec db psql -U wodore -d wodore -c "CREATE EXTENSION IF NOT EXISTS unaccent;"

# Production/Kubernetes
kubectl exec wd-backend-postgres-1 -- psql -U postgres -d wodore -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
kubectl exec wd-backend-postgres-1 -- psql -U postgres -d wodore -c "CREATE EXTENSION IF NOT EXISTS unaccent;"

# Or via any PostgreSQL client
psql -U wodore -d wodore -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
psql -U wodore -d wodore -c "CREATE EXTENSION IF NOT EXISTS unaccent;"

Note: Most managed PostgreSQL services (AWS RDS, Google Cloud SQL, Azure Database) allow these extensions without superuser privileges. If you encounter permission errors, contact your database administrator.

Start Application

Start the application using the app alias (recommended):

(.venv) app migrate
(.venv) app run -p 8000 # -i # with infisical

Or use invoke with infisical:

(.venv) app migrate -i
(.venv) app run -p 8000 -i
(.venv) # or written out
(.venv) inv app.app -i --cmd "migrate"
(.venv) inv app.app -i --cmd "runserver"

Or use local env files (requires .env and config/.env):

(.venv) inv app.app --cmd "migrate"
(.venv) inv app.app --cmd "runserver"

NOTE: The app command expands to if infisical is used:

infisical run --env=dev --path /backend --silent --log-level warn -- app <command>

Load Data

Copy hut information from sources, this saves huts information from different sources (e.g. refuges.info, wikidata, open stree map) into the local database

# Add all available sources
(.venv) app hut_sources --add --orgs all

# Add specific source (e.g. refuges)
(.venv) app hut_sources --add --orgs refuges

Add huts from the previously added sources. If a hut has multiple sources they are combined as good as possible.

# Add huts from sources (combines data if multiple sources)
(.venv) app huts --add-all

Helpful Commands

Common database commands:

# Apply migrations
(.venv) app migrate

# Load initial data
(.venv) app loaddata --app huts organizations

# Squash migrations
(.venv) app squashmigrations huts 0006 --squashed-name init

Watch and compile Tailwind CSS:

npx tailwindcss -i styles.css -o server/apps/manager/static/css/styles.css --minify --watch

Sync Martin tile server assets for production (Kubernetes):

# Preview sync (dry-run) - syncs all categories by default
(.venv) app martin_sync --dry-run

# Sync to default target (./martin_sync)
(.venv) app martin_sync

# Sync specific categories only
(.venv) app martin_sync --include accommodation,transport

# Sync to custom target (e.g., production PVC)
(.venv) app martin_sync --target /mnt/martin-pvc

Package Updates

Update all packages:

(.venv) inv update # OR
(.venv) inv update --no-private # do not update private packages (this removes the private packages)

# Update hut-service (private package only)
(.venv) inv update -p hut-services-private
(.venv) # uv sync --upgrade-package hut-services-private --extra private
(.venv) # uv lock

Changes

After changes the version in pyproject.toml needs to be updated and the wodore-backend package updated and the docker image published:

(.venv) vim pyproject.toml
(.venv) inv update -p wodore-backend
(.venv) # uv sync --upgrade-package wodore-backend --extra private
(.venv) inv docker.build --push # --version-tag

Release

For a release run inv release. Merge this change into the main branch, the github action will create a tag and a release.

Docker Production Build

Set required environment variables (or add it to the .env file):

READ_GITHUB_USER=<username>
READ_GITHUB_TOKEN=<token>  # Must have read access

(run infisical export --env dev --path /keys/wodore-backend to export the secrets)

Build and run Docker images (default is alpine image):

# Build main image
(.venv) inv docker.build [--distro alpine|ubuntu] [-p/--push] [-v/--version-tag]

# Create slim version (optional)
(.venv) inv docker.slim [--distro alpine|ubuntu]

# Run the container
(.venv) inv docker.run [--distro alpine|ubuntu] [--slim]

# Publish the container (use -v to include version tags as well, otherwise only 'edge' is pushed)
(.venv) inv docker.publish [--distro alpine|ubuntu] [--slim] [-v/--version-tag]

NOTE: These commands are deprecated:

# Export secrets (will be removed)
infisical export --env dev --path /backend >> config/.env

# Build staging (use --env=prod for production)
infisical run --env=dev --path /backend -- \
  docker compose -f docker-compose.yml \
  -f docker/docker-compose.stage.yml build web

Prerequisites

Required development tools:

• python3.12 (see pyproject.toml)
• postgresql13
• docker with docker compose
• infisical (installation guide) (optional)
• uv (installation guide)
• node and npm for Tailwind CSS
• make (optional)
• martin tile server (installation guide)

TODOs

See TODOS.md for future improvements and refactoring ideas.