Improve directory route normalization and HTML 404s

[jolestar]

Summary

• redirect extensionless directory homepage requests like /guides to the canonical /guides/
• render a built-in HTML 404 page for human HTML requests
• keep markdown, asset, and API misses machine-friendly

Details

• /foo now returns 308 to /foo/ when foo/ is a valid directory route
• missing HTML page requests now return a rendered site-shell 404 document
• raw markdown 404s still return plain text Not Found
• routing docs updated to describe canonical trailing-slash directory routes and HTML 404 behavior

Verification

• npm run check
• npm test

Closes #30

[Copilot]

Pull request overview

Updates the core request handler to improve UX for directory-backed routes and to serve a human-friendly HTML 404 page, while preserving machine-friendly 404s for markdown/assets/APIs.

Changes:

• Redirects extensionless directory route requests (e.g. /guides) to the canonical trailing-slash form (/guides/) when the directory exists.
• Renders a built-in HTML 404 document (site shell) for unresolved HTML page requests, while keeping markdown 404s plain text.
• Updates routing docs and adds tests for the new behaviors.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/core/request-handler.ts	Adds canonical directory redirects and HTML 404 rendering logic in the core router/request layer.
src/core/request-handler.test.ts	Adds tests for HTML vs markdown 404 behavior and for directory trailing-slash redirects.
docs/site/concepts/routing.md	Documents canonical trailing-slash directory routes and HTML 404 behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

404 responses for negotiable HTML routes vary by Accept (HTML 404 vs plain-text 404 when Accept: text/markdown), but renderNotFoundForResolvedRequest/renderHtmlNotFound currently never set Vary: Accept. This can cause caches/CDNs to serve the wrong 404 representation across clients. Please add Vary: Accept whenever shouldVaryOnAccept(resolved) is true (both for the HTML 404 and the plain-text 404 returned when negotiatedMarkdown is true).

[Copilot]

tryRedirectCanonicalDirectoryPath always returns a redirect once listDirectory() succeeds (line 1126), so the preceding loop that fetches/parses index candidates never affects the outcome and adds unnecessary I/O/CPU. Consider simplifying this to just return redirect(\${resolved.requestPath}/`)after thelistDirectory` null check, or (if draft/index detection was intended) change the final unconditional redirect so the loop meaningfully gates the redirect.

Suggested change

	for (const candidatePath of getDirectoryIndexCandidates(directoryPath)) {
	const entry = await store.get(candidatePath);
	if (entry === null || entry.kind !== 'text' || entry.text === undefined) {
	continue;
	}
	const parsed = await parseMarkdownDocument(candidatePath, entry.text);
	if (parsed.meta.draft === true && options.draftMode === 'exclude') {
	continue;
	}
	return redirect(`${resolved.requestPath}/`);
	}

[Copilot]

This new 404 test doesn't cover the Accept-negotiated behavior that now affects 404s for extensionless HTML routes (HTML vs plain text). Please add coverage asserting Vary: Accept on the 404 response for /missing and a case where acceptHeader: 'text/markdown' yields the plain-text 404 with the same Vary: Accept header, to prevent cache-related regressions.

Suggested change

	const htmlResponse = await handleSiteRequest(new MemoryContentStore([]), '/missing', {
	draftMode: 'exclude',
	siteConfig: TEST_SITE_CONFIG,
	});
	assert.equal(htmlResponse.status, 404);
	assert.equal(htmlResponse.headers['content-type'], 'text/html; charset=utf-8');
	assert.match(String(htmlResponse.body), /<h1>Not Found<\/h1>/);
	assert.match(String(htmlResponse.body), /No page was published at <code>\/missing<\/code>/);
	assert.match(String(htmlResponse.body), /<title>Not Found \| Test Site<\/title>/);
	const markdownResponse = await handleSiteRequest(new MemoryContentStore([]), '/missing.md', {
	draftMode: 'exclude',
	siteConfig: TEST_SITE_CONFIG,
	});
	const store = new MemoryContentStore([]);
	const htmlResponse = await handleSiteRequest(store, '/missing', {
	draftMode: 'exclude',
	siteConfig: TEST_SITE_CONFIG,
	});
	assert.equal(htmlResponse.status, 404);
	assert.equal(htmlResponse.headers['content-type'], 'text/html; charset=utf-8');
	assert.equal(htmlResponse.headers['vary'], 'Accept');
	assert.match(String(htmlResponse.body), /<h1>Not Found<\/h1>/);
	assert.match(String(htmlResponse.body), /No page was published at <code>\/missing<\/code>/);
	assert.match(String(htmlResponse.body), /<title>Not Found \| Test Site<\/title>/);
	const negotiatedMarkdownResponse = await handleSiteRequest(store, '/missing', {
	draftMode: 'exclude',
	siteConfig: TEST_SITE_CONFIG,
	acceptHeader: 'text/markdown',
	});
	assert.equal(negotiatedMarkdownResponse.status, 404);
	assert.equal(negotiatedMarkdownResponse.headers['content-type'], 'text/plain; charset=utf-8');
	assert.equal(negotiatedMarkdownResponse.headers['vary'], 'Accept');
	assert.equal(String(negotiatedMarkdownResponse.body), 'Not Found');
	const markdownResponse = await handleSiteRequest(store, '/missing.md', {
	draftMode: 'exclude',
	siteConfig: TEST_SITE_CONFIG,
	});