xspct_db

A multi-backend database query service with two-layer caching, concurrency control, and Rspamd integration.

Provides an async HTTP API (aiohttp) for querying user data from LDAP, MySQL, and YAML backends, with a two-layer object cache (in-process L1 TTLCache + optional Redis L2), a response-level cache for batch POST endpoints, foreground/background query queues with semaphore-based concurrency control, API-key authentication, Prometheus metrics, and TLS support.

Installation

Requires Python 3.10 or newer.

pip install "xspct_db @ git+https://github.com/HeinleinSupport/xspct_db.git"                   # core (aiohttp + PyYAML)
pip install "xspct_db[ldap] @ git+https://github.com/HeinleinSupport/xspct_db.git"          # + bonsai LDAP support
pip install "xspct_db[mysql] @ git+https://github.com/HeinleinSupport/xspct_db.git"         # + aiomysql support
pip install "xspct_db[redis] @ git+https://github.com/HeinleinSupport/xspct_db.git"         # + Redis caching
pip install "xspct_db[msgpack] @ git+https://github.com/HeinleinSupport/xspct_db.git"       # + msgpack body encoding
pip install "xspct_db[uvloop] @ git+https://github.com/HeinleinSupport/xspct_db.git"        # + uvloop event loop
pip install "xspct_db[all] @ git+https://github.com/HeinleinSupport/xspct_db.git"           # all optional backends
pip install "xspct_db[all,dev] @ git+https://github.com/HeinleinSupport/xspct_db.git"       # + dev/test dependencies
pip install "xspct_db[all,dev,docs] @ git+https://github.com/HeinleinSupport/xspct_db.git"  # + Sphinx documentation

Usage

xspct-db /etc/xspct-db.yml
# or
python -m xspct_db /etc/xspct-db.yml

Configuration is a single YAML file. All keys are optional; see docs/guide/configuration.md for the full reference, including examples for multiple databases with result merging and chained queries with use_result.

HTTP API

Method	Path	Auth	Description
GET	/	—	Health / liveness check
GET	/ping	—	Ping → Pong
GET	/metrics	optional	Prometheus metrics
GET	/v1/query/{user}	API key	Single user lookup
POST	/v1/query-json	API key	Batch user lookup
POST	/v1/rspamd-settings	API key	Rspamd settings blob

Legacy path prefixes (/query/v1/{user}, /query-json/v1, /rspamd-settings/v1) are also accepted for backwards compatibility.

Authentication uses the X-Api-Key header (configurable). See docs/guide/api.md for request/response details and all exposed metrics.

Backends

Backend	Extra	Description
yaml	—	Static data from the config file
dummy	—	No-op backend (returns the username as-is)
delay	—	Artificial-delay backend for testing
ldap	[ldap]	LDAP via bonsai with connection pooling
mysql	[mysql]	MySQL via aiomysql with connection pooling

Caching

xspct_db has three independent cache layers, all using TTLCache from cachetools:

Layer	Config key	Scope
L1 object cache	xspct_db_local_cache	Per-user lookup; zero latency, in-process
L2 object cache	xspct_db_redis_cache	Per-user lookup; shared across workers via Redis
Response cache	xspct_db_response_cache	Full serialised response body for POST /v1/query-json and POST /v1/rspamd-settings (JSON or msgpack, cached separately)

On a GET /v1/query/{user} request, lookups flow: L1 → L2 (Redis) → backend. On a POST request, the response cache is checked first; on a miss the backend is queried and the serialised response is stored for reuse.

Wildcard Domain Query

Any query can be configured with wildcard_domain_query: true. When a user address is not found by the regular lookup, xspct_db re-runs that query using a wildcard key derived from the address (default: strip one subdomain level → user@sub.example.com → @example.com). The fallback result is returned under the original address in the response. When multiple queries enable wildcard fallback, each query derives its wildcard key from its own wildcard_key_pattern / wildcard_key_replacement settings.

The key derivation is configurable per query via wildcard_key_pattern and wildcard_key_replacement (Python re.sub syntax). See docs/guide/configuration.md for the full reference and examples.

Address Rewrite Rules

xspct_db_rewrite_rules rewrites incoming addresses before the prefilter, object cache lookup, and backend queries run. This is useful for relay domains, SASL realms, or other alias forms that should map to a canonical mailbox.

Rules are evaluated in order, and the first rule that actually changes the address wins. The response is still keyed under the original address received from the client, and both the original and canonical forms are registered as cache aliases.

xspct_db_rewrite_rules:
  - pattern: '^(.+)@relay\\.example\\.com$'
    replacement: '\\1@example.com'

See docs/guide/configuration.md for the full reference and examples.

Concurrency

When xspct_db_request_timeout is greater than 0, every query endpoint is guarded by two asyncio.Semaphore instances:

• Foreground slots (xspct_db_foreground_slots, default 30) — limit concurrent client-blocking queries. When all slots are busy a new request immediately receives 503 Service Overloaded.
• Background slots (xspct_db_background_slots, default 5) — when a query exceeds the timeout the client receives 504 Request Timeout but the backend task is promoted to a background slot so it can complete and warm the cache for subsequent requests. If no background slot is free the task is cancelled.

See docs/guide/configuration.md for all options.

Development

git clone https://github.com/HeinleinSupport/xspct_db
cd xspct_db
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov

# Build documentation
pip install -e ".[docs]"
python -m sphinx -b html docs docs/_build/html

License

European Union Public Licence v. 1.2 (EUPL-1.2) — see LICENSES/EUPL-1.2.txt.

Changelog

See CHANGELOG.md.