feat(HNT-2097): scaffold monorepo with deployable healthcheck services

[mmiermans]

Goal

HNT-2097 Scaffold the hnt-content monorepo with two deployable services for the Crawling Pipeline v2.

Implementation Decisions

• ESM-only ("type": "module", moduleResolution: "nodenext") — no CJS compatibility layer
• Vitest instead of Jest — avoids --experimental-vm-modules dependency for ESM testing
• pnpm deploy --prod with injectWorkspacePackages for Docker isolation instead of copying the full workspace
• Crawl-agent healthcheck fails if no tick within a configurable threshold, so K8s restarts stuck pods

[Copilot]

Pull request overview

Scaffolds the new hnt-content monorepo with Turborepo + pnpm workspaces, introduces a shared crawl-common package, and adds two deployable Node/Express services (crawl-agent, crawl-worker) that expose /healthz endpoints (with tick-staleness detection in crawl-agent).

Changes:

• Add Turborepo + pnpm workspace + TypeScript/Vitest + ESLint/Prettier scaffolding for a Node 24 ESM monorepo.
• Introduce crawl-agent and crawl-worker Express services with healthcheck endpoints and basic test coverage.
• Add Dockerfile + GitHub Actions PR workflow for build/test/lint and docs/templates for repo conventions.

Reviewed changes

Copilot reviewed 32 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
turbo.json	Defines Turbo tasks for build/test/lint/dev across the monorepo.
tsconfig.json	Establishes base TypeScript (ESM/NodeNext) compiler settings for the workspace.
package.json	Root workspace scripts (turbo run), tooling deps, Node engine, pnpm pin.
eslint.config.js	Flat-config ESLint setup using typescript-eslint + Prettier config.
pnpm-workspace.yaml	Declares workspace package globs and pnpm workspace settings.
pnpm-lock.yaml	Locks all dependency versions for the new workspace.
Dockerfile	Multi-stage Docker build using turbo prune + pnpm deploy for service images.
README.md	Documents dev workflow, architecture, and Docker build instructions.
CODE_OF_CONDUCT.md	Adds Mozilla Community Participation Guidelines link.
CLAUDE.md	Documents testing/debugging commands and git conventions.
.prettierrc	Prettier formatting rules for the repo.
.nvmrc	Pins local dev Node major version (24).
.gitignore	Ignores node_modules, dist, turbo cache, env files, etc.
.dockerignore	Reduces Docker build context size and avoids copying secrets/dev files.
.github/workflows/pull-request.yml	CI jobs for build/test/lint on PRs.
.github/PULL_REQUEST_TEMPLATE.md	Standardizes PR description format.
packages/crawl-common/package.json	Defines crawl-common package build/test/lint and ESM exports.
packages/crawl-common/tsconfig.json	Package-level TS config (outDir/rootDir) extending root.
packages/crawl-common/tsup.config.ts	TSUP config to build ESM + d.ts for the shared package.
packages/crawl-common/vitest.config.ts	Vitest config for the shared package.
packages/crawl-common/src/index.ts	Initial placeholder entrypoint for shared code.
services/crawl-agent/package.json	Service package definition and scripts for build/dev/test/lint.
services/crawl-agent/tsconfig.json	Service TS config (dist output) extending root.
services/crawl-agent/vitest.config.ts	Vitest config for crawl-agent tests.
services/crawl-agent/src/app.ts	Express app + /healthz endpoint with tick staleness logic.
services/crawl-agent/src/config.ts	Service configuration values (port/tick intervals).
services/crawl-agent/src/main.ts	HTTP server startup + placeholder tick loop + shutdown handling.
services/crawl-agent/src/app.spec.ts	Tests for /healthz success and stale tick failure behavior.
services/crawl-worker/package.json	Service package definition and scripts for build/dev/test/lint.
services/crawl-worker/tsconfig.json	Service TS config (dist output) extending root.
services/crawl-worker/vitest.config.ts	Vitest config for crawl-worker tests.
services/crawl-worker/src/app.ts	Express app + /healthz endpoint.
services/crawl-worker/src/config.ts	Service configuration values (port).
services/crawl-worker/src/main.ts	HTTP server startup + SIGTERM shutdown handling.
services/crawl-worker/src/app.spec.ts	Tests for /healthz endpoint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

staleTickThresholdMinutes (and tickIntervalMs) are hard-coded, but the PR description says the healthcheck threshold is configurable. Consider reading these values from env (with validation and sensible defaults) so the threshold can be tuned per environment without a rebuild/redeploy.

[Copilot]

shutdown() clears the active sleep timeout, but sleep() never resolves/rejects when that happens. If the agent is in the await sleep(...) call during shutdown, the run() loop will remain stuck and never reach the while (isRunning()) check again (which becomes important if you later add cleanup logic after the loop). Make the sleep cancellable (e.g., resolve immediately on shutdown, or use an AbortSignal) so the loop can exit deterministically.

[Copilot]

parseInt(process.env.PORT ?? '8080', 10) can yield NaN for invalid input (e.g. PORT=abc), which will later cause app.listen to throw with a less actionable error. Validate the parsed value (finite integer within 1–65535) and throw a clear startup error if it’s invalid.

Suggested change

	export default {
	port: parseInt(process.env.PORT ?? '8080', 10),
	const rawPort = process.env.PORT ?? '8080';
	const port = Number.parseInt(rawPort, 10);
	if (
	!Number.isFinite(port) ||
	!Number.isInteger(port) ||
	port < 1 ||
	port > 65535
	) {
	throw new Error(
	`Invalid PORT environment variable "${rawPort}". ` +
	'Expected an integer between 1 and 65535.'
	);
	}
	export default {
	port,

[Copilot]

parseInt(process.env.PORT ?? '8080', 10) can yield NaN for invalid input (e.g. PORT=abc), which will later cause app.listen to throw with a less actionable error. Validate the parsed value (finite integer within 1–65535) and throw a clear startup error if it’s invalid.

Suggested change

	export default {
	port: parseInt(process.env.PORT ?? '8080', 10),
	const rawPort = process.env.PORT ?? '8080';
	const parsedPort = Number.parseInt(rawPort, 10);
	if (!Number.isFinite(parsedPort) || parsedPort < 1 || parsedPort > 65535) {
	throw new Error(
	`Invalid PORT environment variable "${rawPort}". Expected an integer between 1 and 65535.`,
	);
	}
	export default {
	port: parsedPort,

[mmiermans]

I don't think this is worth the complexity.

[Copilot]

The Docker image build enables Corepack but never pins/activates the repo’s declared pnpm version (package.json has packageManager: pnpm@10.28.0). This can make Docker builds non-reproducible (Node’s bundled pnpm/Corepack version may differ) and may break pnpm deploy semantics across Node image updates. Activate the pinned pnpm version explicitly (e.g., corepack prepare pnpm@10.28.0 --activate, or copy package.json early and run corepack install).

Suggested change

	RUN corepack enable
	RUN corepack enable
	RUN corepack prepare pnpm@10.28.0 --activate

[Copilot]

The --mount=type=cache,...,target=/pnpm/store cache is likely ineffective because pnpm’s store directory isn’t configured to /pnpm/store (by default it uses a path under the user’s home). Configure the store dir (e.g., pnpm config set store-dir /pnpm/store or ENV PNPM_STORE_DIR=/pnpm/store) so the cache mount is actually used.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 35 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

sleep() adds a new abort event listener on every tick and never removes it. Over time this will accumulate listeners and can cause memory growth / MaxListeners warnings in a long-running pod. Register the handler with { once: true } and/or remove the listener after it fires.

Suggested change

	});
	}, { once: true });

[Copilot]

Only SIGTERM is handled here. For local dev (Ctrl+C) and some orchestrators, SIGINT is also common. Consider wiring the same shutdown path for SIGINT, and (similar to other services) consider observing the server.close callback to ensure graceful termination.

[Copilot]

pnpm run build --filter=${SCOPE}... likely passes --filter through to the build script instead of applying pnpm's workspace filter, which can break the Docker build. Use pnpm's filter syntax (e.g. place --filter before the command) so only the selected scope (and deps) is built.

Suggested change

	RUN pnpm run build --filter=${SCOPE}...
	RUN pnpm --filter=${SCOPE}... run build