# Monk Search Gateway

A modular NestJS + GraphQL Yoga service that exposes a GraphQL DSL translating to Apache Solr and Elasticsearch queries. Configuration (mappings, sinks, and virtual fields) is sourced from etcd and hot-reloaded via native watchers, enabling dynamic routing and query expansion.

## Features

- **GraphQL DSL** (built with Pothos) for rich boolean queries, term/phrase/wildcard clauses, ranges, and virtual fields.
- **Virtual field expansion** with recursive templating, validated against registry-provided configuration.
- **Search routing** across Solr and Elasticsearch sinks with field mapping resolution per sink tag priority.
- **Result aggregation** including score min-max normalization and aggregation merging.
- **Effect.ts-powered orchestration** for async workflows and error handling.
- **Observability** via Envelop OpenTelemetry and Prometheus plugins, leveraging NestJS built-in logging.
- **Configuration management** backed by etcd, JSON Schema validation (Ajv), and hot reload support through etcd watch streams.
- **Tooling**: Vitest for tests, Biome for lint/format, graphql-codegen, and tsx for development.

## Project structure

```
src/
  app/                # Nest root module
  aggregation/        # Score normalization and aggregation merging
  config/             # Datastore abstraction, JSON schema validation, snapshots
  graphql/            # Pothos schema builder, DSL parsing, module wiring
  observability/      # Envelop OpenTelemetry & Prometheus plugins
  query/              # DSL AST, translators, virtual field handling
  search/             # Sink routing, engine payload generation, clients
```

## Getting started

1. **Install dependencies**

   ```bash
   pnpm install
   ```

2. **Environment configuration**

   Set the following variables (defaults shown):

   ```bash
   export ETCD_ENDPOINTS=http://127.0.0.1:2379
   export ETCD_NAMESPACE=/monk
   export CONFIG_MAPPINGS_KEY=/monk/config/mappings
   export CONFIG_SINKS_KEY=/monk/config/sinks
   export CONFIG_VIRTUAL_FIELDS_KEY=/monk/config/virtual_fields
   export CONFIG_FALLBACK_DIR=./config
   # Optional: ETCD_USERNAME / ETCD_PASSWORD for authenticated clusters
   # Optional: CONFIG_POLL_INTERVAL_MS for periodic refresh as a safety net
   ```

   Sample fallback payloads live under `config/` (`config/mappings.json`, `config/sinks.json`, and `config/virtual.json`). They are loaded automatically if the datastore is unavailable.

3. **Run the service**

   ```bash
   pnpm dev
   # or build & start
   pnpm build
   node dist/main.js
   ```

4. **GraphQL Playground**

   Yoga exposes the GraphQL endpoint on `http://localhost:3000/graphql` with the schema defined in `schema.graphql` (kept in sync with the Pothos builder).

## Quality checks

```bash
pnpm lint     # biome linting
pnpm format   # biome formatting
pnpm test     # vitest unit tests
pnpm generate # graphql-codegen (types -> src/graphql/types/generated.ts)
```

## Testing

- `VirtualFieldService` unit tests validate recursive expansion, templating, and recursion detection. Extend the Vitest suite with translator/routing scenarios as functionality grows.

## Extensibility

- **New clause types**: implement a translator in `src/query/clauses`, register it in both Solr and Elasticsearch registries, and update the GraphQL inputs plus parser.
- **Additional sinks**: extend `SearchClientFactory` with a new client adapter and enrich the JSON Schemas if new configuration fields are needed.
- **Observability**: register additional Envelop plugins via `ObservabilityModule` and wire exporters as needed.
- **Configuration**: add JSON Schemas in `schema.graphql` and update Ajv validation logic when registry payloads evolve.

## Notes

- etcd watches stream live change events; `CONFIG_POLL_INTERVAL_MS` can enable periodic refreshes as a safety net.
- Local fallback configuration is sourced from the JSON files in `config/`; override the path with `CONFIG_FALLBACK_DIR`.
- Search clients are stubs that log the intended payload. Replace with real HTTP clients (e.g., `@elastic/elasticsearch`, `solr-client`) and enrich error handling and retry policies.
- Aggregation merging currently collates engine-specific payloads side-by-side; plug in combination logic as requirements solidify.