NCEAi (Backend Core)

This is the backend task-processing and API-serving engine for NCEAi, an AI study platform for New Zealand NCEA students. The backend manages PDF syllabus document ingestion, builds curriculum-aligned multiple-choice quizzes using structured LLM outputs, tracks student assessment attempts, and handles user authentication.

The codebase is split into a Flask web server and a multi-threaded background task service connected via Redis and PostgreSQL. The corresponding frontend repository can be found here.

Directory Structure

├── api.py                   # Flask API entry point
├── task_service.py          # Multi-threaded background task consumer
├── configs/                 # YAML files for database, redis, task, and quiz settings
├── sql/                     # PostgreSQL database schema and migrations
├── api_utils/               # Flask routing, decorators, and controllers
│   ├── routes.py            # API endpoint definitions
│   ├── decorators.py        # Idempotency caching decorator
│   └── services/            # Handlers for auth, user, quiz, and file logic
├── nceai_utils/             # Shared helper utilities
│   ├── auth_helpers/        # JWT utilities and refresh token storage logic
│   ├── redis_helpers/       # Redis connection clients
│   └── task_helpers/        # Task producers, consumers, and rate-limiting locks
└── quiz_generation/         # LLM pipeline and structured schemas
    ├── generator.py         # Multi-step generation orchestrator
    └── schemas/             # Pydantic validation models for generation outputs

System Implementation

Background Task Runner (task_service.py)

Since generating high-quality questions takes significant time, all generation jobs are offloaded to background workers:

• Task Queue: Tasks are queued in Redis using a blocking FIFO pattern (BLPOP) to wake worker threads without polling.
• Concurrency Control: Tasks are processed by a pool of daemon threads limited by a semaphore to protect system resources.
• PostgreSQL Row-Locking: To prevent race conditions, the worker locks user rows (SELECT FOR UPDATE) before checking if they have exceeded their concurrent task limit.
• Auto-Recovery: If a task fails or throws an exception, the system catches it, backs off for 20 seconds, and requeues it in Redis.

Multi-Stage Assessment Pipeline (quiz_generation/)

Rather than asking an LLM to generate an entire quiz in one prompt, the pipeline is divided into distinct, stateful stages:

1. Context Generation
2. Concept Extraction
3. Concept Curriculum Alignment
4. Stem (Question) Generation
5. Correct Option Generation
6. Distractor (Decoy Answer) Generation
7. Explanation Generation

Each stage validates its output against structural Pydantic models. The pipeline can query local vLLM or Ollama instances, or cloud endpoints like the OpenAI API. Text extraction from uploaded PDFs is handled using IBM's Docling parser.

API Gateway & Session Security (api_utils/)

• Idempotency Caching: A custom Flask decorator validates an Idempotency-Token header and caches response data in Redis to prevent duplicate generation task creation during client-side retries.
• JWT Authentication: Authentication uses short-lived JWT access tokens and long-lived HTTP-only cookie refresh tokens tracked in Redis. This allows for simple token validation and a stateful force-logout mechanism.

Getting Started

Prerequisites

• PostgreSQL
• Redis
• Python 3.10+

Environment Setup

Create a .env file in the root directory:

DB_PASSWORD=your_postgres_password
REDIS_PASSWORD=your_redis_password
ACCESS_SECRET_KEY=your_jwt_access_secret
REFRESH_SECRET_KEY=your_jwt_refresh_secret
OPENAI_API_KEY=your_openai_api_key
VLLM_API_KEY=EMPTY
VLLM_API_BASE_URL=http://localhost:8000/v1

Database Initialization

psql -d nceai_db -f sql/001_create_nceai_app_schema.sql

Running the Services

1. Start the Flask API:

   python api.py

2. In a separate terminal, start the task consumer daemon:

   python task_service.py