Production-readiness gaps and roadmap for a genuinely AI-powered recommender

[CoreyLeath-code]

Executive summary

DeepSequence-Recommender has useful building blocks—a PyTorch sequence model, FastAPI endpoints, Prometheus metrics, containers, and tests—but it is not production-ready yet. Most importantly, the deployed API does not load a trained recommender: every startup builds a 200-item demo vocabulary and initializes fresh random model weights.

This means the service is technically running neural-network inference, but its recommendations are not learned from user behavior and are not meaningful or reproducible across restarts.

Observed weaknesses

1. Production serves an untrained, randomly initialized model

app/main.py creates DeepSequenceModel at startup but never loads a trained checkpoint, item vocabulary, feature schema, or model metadata. The in-memory vocabulary is generated from item_0 through item_199.

Consequences:

• recommendations change when the process restarts;
• no offline training result reaches production;
• real catalogue items outside the demo vocabulary become padding/unknown ID 0;
• model and vocabulary versions cannot be audited or rolled back.

2. There are two disconnected recommender implementations

The FastAPI path uses app/core/model.py. The separate recommender/orchestrator.py sleeps for 15 ms and generates random IDs/scores, and it is not the model used by the API. The ONNX exporter serializes a third MockSequentialModel, not the API's DeepSequenceModel.

These parallel implementations create documentation drift and do not form one tested training-to-serving path.

3. Ranking correctness has unresolved edge cases

The vocabulary assigns real items IDs 1..num_items, but the output layer has num_items classes with indices 0..num_items-1. As written:

• the last vocabulary item cannot be predicted;
• class 0 can be selected even though it represents padding and decodes to None;
• attention does not mask left-padding, so padding states affect the context vector;
• top_k has no API bounds and can exceed the output dimension;
• empty and entirely unknown sequences are accepted as successful inference.

These are model-quality and API-contract problems, not only polish items.

4. Benchmark and availability claims are not currently reproducible

The README advertises approximately 12,000 requests/second, Triton/ONNX serving, 15–42 ms p99 latency, 95% latency compression, and a 99.99% availability guarantee. The checked-in container actually runs a single Uvicorn command over PyTorch, and no evidence artifact ties those figures to a commit, hardware profile, dataset, load shape, or statistical method.

The benchmark workflow runs:

pytest --benchmark-only --benchmark-skip || echo "Performance profiles logged successfully."

This can pass when benchmarks fail or do not exist. The main CI test command also ends with || echo, allowing test failures to produce a successful job.

Targets should be labeled as targets until they are measured and enforced.

5. No recommender-quality evaluation exists

Unit tests validate shapes and API responses, but there is no offline evaluation using:

• temporal train/validation/test splits;
• Recall@K / HitRate@K;
• NDCG@K;
• MRR@K;
• catalogue coverage and novelty;
• diversity and popularity bias;
• cold-start cohorts;
• comparison against popularity, item-item, and matrix-factorization baselines.

Without ranking metrics and baselines, there is no evidence that the neural model improves recommendations.

6. Missing ML lifecycle and data contracts

The repository lacks one end-to-end, reproducible path for:

• interaction-event validation;
• deterministic dataset snapshots;
• sessionization and negative sampling;
• training and hyperparameter tracking;
• vocabulary/feature artifact persistence;
• model registry promotion;
• checkpoint signing and compatibility checks;
• shadow, canary, rollback, and retraining policies;
• training-serving skew and drift detection.

7. Serving architecture is not ready for production load

The request handler performs synchronous CPU inference against one process-global model. There is no dynamic batching, queue backpressure, concurrency/load test, timeout budget, graceful shutdown/drain, model warmup gate, or capacity plan.

Redis settings and cache metrics exist, but the request path does not implement a Redis cache. The fallback logic described in the README belongs to the disconnected mock orchestrator and is not protecting the FastAPI endpoint.

8. Security, privacy, and supply-chain gaps

• secret_key defaults to change-this-secret instead of failing closed in production.
• The API has no authentication, authorization, rate limiting, request-size limits, or abuse controls.
• Dependencies use broad lower bounds rather than a reproducible lock/constraints file.
• User identifiers and click histories lack retention, deletion, consent, and log-redaction policy.
• The container has no explicit health check and retains build tooling through its base stage.

How to make it genuinely AI-powered

The repository already uses PyTorch; the next step is to turn model-shaped code into a trained, evaluated, feedback-driven recommender.

Core recommendation stack

1. Build an event contract for user, item, event type, timestamp, session, context, and schema version.
2. Create temporal dataset snapshots with deterministic sessionization and leakage-safe splits.
3. Establish baselines: global popularity, recent popularity, item-item co-occurrence, and implicit matrix factorization.
4. Train the sequence model with next-item targets, sampled negatives, reproducible seeds, and tracked experiments.
5. Persist a versioned bundle containing weights, item mapping, feature schema, hyperparameters, metrics, and data lineage.
6. Load that bundle at startup and refuse readiness when artifacts are missing or incompatible.
7. Evaluate Recall@K, NDCG@K, MRR@K, coverage, diversity, novelty, calibration, and cohort performance.
8. Promote models only when they outperform declared baselines and pass latency/resource gates.

Higher-value AI features

• Hybrid retrieval and ranking: two-tower candidate retrieval followed by the sequence model as a ranker.
• Content-aware cold start: text/image embeddings for new items and preference initialization for new users.
• Contextual ranking: time, device, locale, inventory, price, and session-intent features.
• Multi-objective re-ranking: balance relevance with diversity, freshness, availability, margin, and policy constraints.
• Exploration: contextual bandits with guardrails to learn from online feedback.
• Natural-language discovery: use an LLM for query/intent understanding and explanations, while the recommender remains the source of ranking truth.
• Explanation layer: cite concrete behavioral/content signals; never claim unsupported causal reasons.
• Feedback loop: capture impressions, positions, clicks, skips, dwell time, conversions, and explicit dislikes.

LLMs should enhance intent and explanations, not replace measurable retrieval/ranking models.

Production-readiness roadmap

P0 — Correctness and honest evidence

• Fix vocabulary/output indexing and mask padding in attention.
• Validate non-empty known-item sequences and constrain 1 <= top_k <= catalogue_size.
• Remove random/mock implementations from production paths or clearly isolate them as examples.
• Make CI and benchmark failures fail the workflow; remove || echo.
• Relabel unsupported README performance figures as targets or replace them with reproducible artifacts.
• Add checkpoint + vocabulary loading and a readiness probe that verifies model availability.
• Add deterministic model-output, restart-reproducibility, invalid-input, and artifact-compatibility tests.

P1 — Reproducible ML system

• Add versioned schemas, dataset snapshots, temporal splits, training CLI, seeds, and experiment tracking.
• Add baseline implementations and an offline evaluation report.
• Register model bundles with lineage, quality gates, and rollback metadata.
• Test training-serving feature parity and vocabulary compatibility.
• Export and validate the actual production model to ONNX before enabling ONNX serving.

P2 — Reliable serving

• Add async admission control, timeouts, bounded queues, batching, and graceful shutdown.
• Implement a real fallback path and version-aware cache with measurable hit/miss behavior.
• Run reproducible load tests with hardware, concurrency, payload, warmup, sample count, p50/p95/p99, errors, CPU, and memory recorded.
• Define SLOs from observed capacity and add alerting/runbooks.
• Add canary/shadow deployment, automated rollback, and model warmup.

P3 — Online learning and governance

• Add impression/outcome logging and A/B experimentation with guardrail metrics.
• Monitor feature drift, prediction drift, cold-start quality, bias, coverage, and business outcomes.
• Add privacy retention/deletion controls, consent boundaries, audit logs, and PII redaction.
• Add authentication, authorization, rate limiting, secret management, dependency locking, SBOM, and image scanning.

Definition of done

The service should be called production-ready only when:

• a versioned trained model and matching vocabulary are loaded reproducibly;
• offline quality beats declared baselines on leakage-safe temporal evaluation;
• CI cannot hide test or benchmark failures;
• latency/capacity claims are backed by commit-linked artifacts;
• readiness reflects real model and dependency health;
• rollback and fallback behavior are tested;
• online metrics cover quality, reliability, drift, and business outcomes; and
• security/privacy controls are documented and enforced.