diff --git a/.env.example b/.env.example index 1ddea83..61ff416 100644 --- a/.env.example +++ b/.env.example @@ -1,8 +1,23 @@ -GITHUB_TOKEN=your_token -GITHUB_USERNAMES=your_username -GITHUB_ORGS=your_organization -IGNORED_REPOS=repo1,repo2 +# ===== Global Settings ===== +# GITHUB_TOKEN=your_token +# IGNORED_REPOS=repo1,repo2 +# ALLOWED_REFERERS=your-builder-domain.com +# Personal and Hosted instances are mutually exclusive. + +# ===== Personal Instance ===== +# Render a chart of your own accounts. # Advanced: JSON array with optional per-entry tokens + +# GITHUB_USERNAMES=your_username +# GITHUB_ORGS=your_organization + # GITHUB_USERNAMES=["your_username", {"name": "other_username", "token": "github_pat_"}] # GITHUB_ORGS=["your_org", {"name": "private_org", "token": "github_pat_"}] + +# ===== Hosted instance ===== +# Serves charts for others' public repos via ?source=[,...]. +# Comma-separated account names to allow, or * to allow any source. +# WARNING: * spends your GITHUB_TOKEN's rate limit on everyone who uses your instance. + +# GITHUB_ALLOWED_SOURCES= diff --git a/README.md b/README.md index fe3a7f4..d371f69 100644 --- a/README.md +++ b/README.md @@ -44,11 +44,13 @@ Deployable **GitHub language chart generator** — embeddable SVGs for READMEs a - **Theming**: Supports `default`, `light`, and `dark` themes. - **Custom Colours**: Set background (`bg`), text (`text`), and individual language colours (`c1`-`c16`) via query parameters. - **Dynamic Layout:** The legend automatically shifts to a **two-column layout** when displaying 9 or more languages. +- **Automatically fetches GitHub repositories:** Public user and organization sources are automatically fetched; private repos can be fetched if tokens have access (personal mode only). +- **Hosted instances:** Serve charts for others via `?source=` — an enumerated allowlist, or `*` for any GitHub account. Hosted modes fetch public repos only. - **Global Token:** Optional deployment-wide default token, applied to any source without its own `token`, boosting limits from 60 req/hr to 5,000/hr. -- **Automatically fetches GitHub repositories:** Public user and organization sources are automatically fetched; private repos can be fetched if tokens have access. -- **Automatically ignores forks**. -- **Ignored repositories**: Repos ignored via env (`IGNORED_REPOS`). - **Hourly caching**: Used to reduce API calls and improve performance. On fetch failure the last good chart is served and refresh retries after either 5 minutes, or when GitHub rate-limit timer resets. +- **Automatically ignores forks**. +- **Ignored repositories**: Repos ignored via optional (`IGNORED_REPOS`) env var. +- **Preview-only deployments:** Optional `ALLOWED_REFERERS` restricts rendering to your own site — blocks README embeds and hotlinking. ## Customize Your Charts Prefer a visual workflow? Use the [@gh-top-languages/builder](https://github.com/gh-top-languages/builder) to preview themes, colours, and layout options interactively, then easily copy the generated embed URL to quickly deploy. @@ -74,6 +76,7 @@ Prefer a visual workflow? Use the [@gh-top-languages/builder](https://github.com | :--- | :--- | :--- | :--- | :--- | | `test` | Boolean | Uses sample data instead of fetching from GitHub API. | `false` | `?test=true` | | `error` | String | Forces an error SVG with the given message. For testing only. | — | `?error=test` | +| `source` | String | Hosted instances only: comma-separated GitHub account names to chart (max 10 when `GITHUB_ALLOWED_SOURCES=*`). | — | `?source=torvalds,rails` | #### Query Parameters Full parameter reference lives in the [lib README](https://github.com/gh-top-languages/lib#query-parameters). @@ -94,11 +97,19 @@ npm install ``` ### Configuration -Copy `.env.example` to `.env`, and update the variables. -- `GITHUB_USERNAMES`: GitHub usernames to fetch repositories from. Accepts a single value (`masonlet`), comma-separated (`masonlet,secondlet`), or a JSON array with optional per-user tokens (`["masonlet", {"name": "other", "token": "github_pat_..."}]`). -- `GITHUB_ORGS`: GitHub organization names to fetch repositories from. Accepts a single value (`gh-top-languages`), comma-separated (`gh-top-languages,starweb-libs`), or a JSON array with optional per-org tokens (`["gh-top-languages", {"name": "starweb-libs", "token": "github_pat_..."}]`) +Copy `.env.example` to `.env`, and update the variables. Configure exactly one mode: **Personal** or **Hosted**. + +#### Global - `IGNORED_REPOS`: Optional comma-separated repo names to exclude from the chart. Accepts a bare name (`repo`, excluded from all sources) or `owner/name` (`org/repo`, scoped to one source). - `GITHUB_TOKEN`: Optional deployment-wide default token, applied to any source without its own `token`. A per-source token always takes precedence; the global token only authenticates public repos (no private repos) while boosting limits (5,000 req/hr instead of 60/hr). +- `ALLOWED_REFERERS`: Optional comma-separated hostnames. When set, charts render only for requests whose `Origin`/`Referer` hostname matches: for preview-only deployments (e.g. serving a builder site). Blocks README embeds, hotlinks, and direct navigation. + +#### Personal +- `GITHUB_USERNAMES`: GitHub usernames to fetch repositories from. Accepts a single value (`masonlet`), comma-separated (`masonlet,secondlet`), or a JSON array with optional per-user tokens (`["masonlet", {"name": "other", "token": "github_pat_..."}]`). +- `GITHUB_ORGS`: GitHub organization names to fetch repositories from. Accepts a single value (`gh-top-languages`), comma-separated (`gh-top-languages,starweb-libs`), or a JSON array with optional per-org tokens (`["gh-top-languages", {"name": "starweb-libs", "token": "github_pat_..."}]`) + +#### Hosted +- `GITHUB_ALLOWED_SOURCES`: Comma-separated account names selectable via `?source=` (bare URL renders their combined chart), or `*` to allow any GitHub account (`?source=` required, max 10 names per request). Hosted instances chart **public repositories only**; per-source tokens don't exist in this mode. **Warning:** with `*`, anyone can use your instance, and every request they make spends your GITHUB_TOKEN's rate limit. ### Running Locally Your endpoint will be available at http://localhost:3000/api/languages (or your configured PORT) @@ -123,18 +134,27 @@ npm start ``` Or, deploy with Vercel: +**Personal** instance (your own chart): + +[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/gh-top-languages/api&env=GITHUB_USERNAMES&envDescription=Your%20GitHub%20username%28s%29%2C%20comma-separated.%20Optional%20vars%20%28GITHUB_ORGS%2C%20GITHUB_TOKEN%2C%20IGNORED_REPOS%29%20are%20documented%20at%20the%20link%20and%20can%20be%20added%20in%20Vercel%20project%20settings.&envLink=https%3A%2F%2Fgithub.com%2Fgh-top-languages%2Fapi%23configuration) + +**Hosted** instance (serve charts for others via `?source=`): -[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/gh-top-languages/api&env=GITHUB_USERNAMES,GITHUB_ORGS,GITHUB_TOKEN,IGNORED_REPOS&envDescription=GITHUB_USERNAMES%20and%2For%20GITHUB_ORGS%3A%20GitHub%20users%2Forgs%20to%20fetch%20repos%20from.%20GITHUB_TOKEN%3A%20optional%20global%20fallback%20token%20for%20sources%20without%20their%20own.%20IGNORED_REPOS%3A%20optional%20comma-separated%20repo%20names%20to%20exclude%20%28bare%20name%20or%20owner%2Fname%29.&envLink=https%3A%2F%2Fgithub.com%2Fgh-top-languages%2Fapi%23configuration) +[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/gh-top-languages/api&env=GITHUB_ALLOWED_SOURCES&envDescription=Account%20names%20to%20allow%2C%20comma-separated%2C%20or%20*%20for%20any%20account.%20Strongly%20recommended%3A%20add%20GITHUB_TOKEN%20in%20Vercel%20settings%20%28*%20mode%20is%20nearly%20unusable%20at%2060%20req%2Fhr%20without%20one%29.%20See%20link%20for%20all%20options.&envLink=https%3A%2F%2Fgithub.com%2Fgh-top-languages%2Fapi%23configuration) ## Error Responses All errors return HTTP 200 with an error SVG so they render in GitHub README embeds. Common error messages: -- `At least one of GITHUB_USERNAMES or GITHUB_ORGS must be set` — missing environment configuration -- `GITHUB_USERNAMES/GITHUB_ORGS must be a valid JSON array. Check your configuration.` — malformed JSON array in env config -- `GitHub API error: {status} {statusText}` — GitHub API request failed -- `No language data available` — no public repositories found +- `Set GITHUB_USERNAMES/GITHUB_ORGS, or GITHUB_ALLOWED_SOURCES for a hosted instance`: missing environment configuration +- `GITHUB_ALLOWED_SOURCES cannot be combined with GITHUB_USERNAMES/GITHUB_ORGS`: both modes configured at once +- `Source selection is not enabled on this instance` / `Unknown or disallowed source` / `Invalid source name` / `Too many sources`: bad `?source=` request; these are CDN-cached for 5 minutes +- `Unknown GitHub account`: `?source=` named an account that doesn't exist (cached 10 minutes) +- `This instance only serves its own site - deploy your own from github.com/gh-top-languages/api`: request blocked by `ALLOWED_REFERERS` (cached 5 minutes) +- `GITHUB_USERNAMES/GITHUB_ORGS must be a valid JSON array. Check your configuration.`: malformed JSON array in env config +- `GitHub API error: {status} {statusText}`: GitHub API request failed +- `No language data available`: no public repositories found - Rate limiting (`403`/`429` with exhausted quota) is detected separately and logged as `GitHub rate limit exceeded; resets at HH:MM:SS`; the cached chart is served until the reset. ## License diff --git a/api/languages/index.ts b/api/languages/index.ts index 0daa1bf..1ceda93 100644 --- a/api/languages/index.ts +++ b/api/languages/index.ts @@ -2,7 +2,7 @@ import type { VercelRequest, VercelResponse } from "@vercel/node"; import { handleLanguages } from "../../src/handler.js"; export default async function handler(req: VercelRequest, res: VercelResponse): Promise { - const { status, headers, body } = await handleLanguages(req.query); + const { status, headers, body } = await handleLanguages(req.query, req.headers); for (const [key, value] of Object.entries(headers)) res.setHeader(key, value); res.status(status).send(body); } diff --git a/src/github/constants.ts b/src/github/constants.ts new file mode 100644 index 0000000..2ef275e --- /dev/null +++ b/src/github/constants.ts @@ -0,0 +1,5 @@ +export const MAX_OPEN_SOURCES = 10; +export const VALID_LOGIN = /^[a-zA-Z0-9](?:-?[a-zA-Z0-9]){0,38}$/; + +export const REFRESH_INTERVAL = 1000 * 60 * 60; +export const NEGATIVE_TTL = 1000 * 60 * 10; diff --git a/src/github/fetch.ts b/src/github/fetch.ts index bd303d4..3ab39ec 100644 --- a/src/github/fetch.ts +++ b/src/github/fetch.ts @@ -1,19 +1,25 @@ +import type { + CacheEntry, + FetchOutcome, + LanguageBytes, + Repo, + Source, + SourceKind +} from "./types.js"; +import { NEGATIVE_TTL, REFRESH_INTERVAL } from "./constants.js"; import { parseNextLink, parseSources } from "./parse.js"; import { FALLBACK_RETRY_MS, rateLimitReset } from "./rateLimit.js"; -import type { LanguageBytes } from "./types.js"; +export class SourceNotFoundError extends Error { + constructor() { super("Unknown GitHub account"); } +} -const REFRESH_INTERVAL = 1000 * 60 * 60; - -let cachedLanguageData: LanguageBytes | null = null; -let lastRefresh = 0; -let inFlightFetch: Promise | null = null; +const cache = new Map(); -type Repo = { - name: string; - fork: boolean; - full_name: string; -}; +export async function fetchSelectedSources(names: string[], strict: boolean): Promise { + const results = await Promise.all(names.map(n => fetchSource("user", { name: n }, strict))); + return mergeLanguages(results); +} export async function fetchLanguageData(useTestData = false): Promise { if (useTestData) { @@ -21,22 +27,6 @@ export async function fetchLanguageData(useTestData = false): Promise { inFlightFetch = null; }); - return inFlightFetch; -} - -export function resetCache(): void { - cachedLanguageData = null; - lastRefresh = 0; - inFlightFetch = null; -} - -async function fetchAndAggregate(now: number): Promise { - const globalToken = process.env["GITHUB_TOKEN"]?.trim() || undefined; const usernames = parseSources(process.env["GITHUB_USERNAMES"]); const orgs = parseSources(process.env["GITHUB_ORGS"]); @@ -44,48 +34,81 @@ async function fetchAndAggregate(now: number): Promise { "At least one of GITHUB_USERNAMES or GITHUB_ORGS must be set" ); + const dedupe = (sources: Source[]) => { + const seen = new Set(); + return sources.filter(s => { + const k = s.name.toLowerCase(); + return seen.has(k) ? false : (seen.add(k), true); + }); + }; + + const results = await Promise.all([ + ...dedupe(usernames).map(u => fetchSource("user", u)), + ...dedupe(orgs).map (o => fetchSource("org", o)) + ]); + + return mergeLanguages(results); +} + +export function resetCache(): void { cache.clear(); } + +async function fetchSource(kind: SourceKind, source: Source, strict = false): Promise { + const key = `${kind}:${source.name.toLowerCase()}`; + let entry = cache.get(key); + if (!entry) { entry = { data: null, lastRefresh: 0, inFlight: null }; cache.set(key, entry); } + + const now = Date.now(); + if (entry.missingUntil && now < entry.missingUntil) { + if (strict) throw new SourceNotFoundError(); + return {}; + } + if (entry.data && now - entry.lastRefresh < REFRESH_INTERVAL) return entry.data; + + const outcome = await (entry.inFlight ??= fetchOne(kind, source, entry, now).finally(() => { entry.inFlight = null; })); + if (outcome.kind === "missing") { + if (strict) throw new SourceNotFoundError(); + console.error(`Skipping ${kind} "${source.name}": account not found.`); + return {}; + } + if (outcome.kind === "failed" && strict) { + const msg = outcome.retryAt + ? `GitHub rate limit exceeded; try again at ${new Date(outcome.retryAt).toLocaleTimeString()}` + : "GitHub API error: source fetch failed"; + throw new Error(msg); + } + return outcome.data; +} + +async function fetchOne(kind: SourceKind, source: Source, entry: CacheEntry, now: number): Promise { + const token = source.token ?? (process.env["GITHUB_TOKEN"]?.trim() || undefined); + let rateLimitResetAt: number | null = null; const noteReset = (ms: number) => { rateLimitResetAt = Math.max(rateLimitResetAt ?? 0, ms); }; - let hadFetchFailure = false; - const repoGroups = await Promise.all([ - ...usernames.map(async u => { - const token = u.token ?? globalToken; - try { - const repos = await fetchAllRepos( - u.token ? `https://api.github.com/user/repos?per_page=100&visibility=all&affiliation=owner` - : `https://api.github.com/users/${encodeURIComponent(u.name)}/repos?per_page=100`, - noteReset, - token - ); - return { token, repos }; - } catch { - hadFetchFailure = true; - console.error(`Skipping user "${u.name}": failed to fetch repositories.`); - return { token, repos: [] as Repo[] }; - } - }), - ...orgs.map(async o => { - const token = o.token ?? globalToken; - try { - const repos = await fetchAllRepos( - `https://api.github.com/orgs/${encodeURIComponent(o.name)}/repos?per_page=100`, - noteReset, - token - ); - return { token, repos }; - } catch { - hadFetchFailure = true; - console.error(`Skipping org "${o.name}": failed to fetch repositories.`); - return { token, repos: [] as Repo[] }; - } - }) - ]); + + let repos: Repo[] = []; + try { + repos = await fetchAllRepos( + kind === "org" ? `https://api.github.com/orgs/${encodeURIComponent(source.name)}/repos?per_page=100` + : source.token ? `https://api.github.com/user/repos?per_page=100&visibility=all&affiliation=owner` + : `https://api.github.com/users/${encodeURIComponent(source.name)}/repos?per_page=100`, + noteReset, + token + ); + } catch (e) { + if (e instanceof SourceNotFoundError) { + entry.missingUntil = now + NEGATIVE_TTL; + entry.data = null; + return { kind: "missing" }; + } + hadFetchFailure = true; + console.error(`Skipping ${kind} "${source.name}": ${(e as Error).message}`); + } const ignored = process.env["IGNORED_REPOS"]?.split(',').map(s => s.trim()) || []; - const languageFetches = repoGroups.flatMap(({ token, repos }) => - repos.filter(repo => !repo.fork && !ignored.includes(repo.name) && !ignored.includes(repo.full_name)).map(repo => + const langResults = await Promise.all( + repos.filter(r => !r.fork && !ignored.includes(r.name) && !ignored.includes(r.full_name)).map(repo => fetch(`https://api.github.com/repos/${repo.full_name.split('/').map(encodeURIComponent).join('/')}/languages`, makeOptions(token)) .then(r => { if (r.ok) return r.json() as Promise; @@ -98,33 +121,26 @@ async function fetchAndAggregate(now: number): Promise { ) ); - const langResults: LanguageBytes[] = await Promise.all(languageFetches); - - const result = langResults.reduce((acc, languages) => { - for (const [lang, bytes] of Object.entries(languages)) { - acc[lang] = (acc[lang] || 0) + bytes; - } - return acc; - }, {}); + const result = mergeLanguages(langResults); if (hadFetchFailure) { if (rateLimitResetAt !== null) console.error( `GitHub rate limit exceeded; resets at ${new Date(rateLimitResetAt).toLocaleTimeString()}` ); - - if (cachedLanguageData !== null) { + if (entry.data !== null) { const retryDelay = rateLimitResetAt ? Math.min(Math.max(rateLimitResetAt - now, 60_000), REFRESH_INTERVAL) : FALLBACK_RETRY_MS; - lastRefresh = now - REFRESH_INTERVAL + retryDelay; - return cachedLanguageData; + entry.lastRefresh = now - REFRESH_INTERVAL + retryDelay; + return { kind: "ok", data: entry.data }; } - return result; + return { kind: "failed", data: result, retryAt: rateLimitResetAt }; } - cachedLanguageData = result; - lastRefresh = now; - return result; + entry.data = result; + entry.lastRefresh = now; + delete entry.missingUntil; + return { kind: "ok", data: result }; } async function fetchAllRepos( @@ -139,6 +155,7 @@ async function fetchAllRepos( while (nextUrl) { const response = await fetch(nextUrl, options); if (!response.ok) { + if (response.status === 404 && repos.length === 0) throw new SourceNotFoundError(); const reset = rateLimitReset(response); if (reset) onRateLimit(reset); throw new Error(`GitHub API error: ${response.status} ${response.statusText}`); @@ -153,6 +170,13 @@ async function fetchAllRepos( return repos; } +function mergeLanguages(all: LanguageBytes[]): LanguageBytes { + return all.reduce((acc, languages) => { + for (const [lang, bytes] of Object.entries(languages)) acc[lang] = (acc[lang] || 0) + bytes; + return acc; + }, {}); +} + function makeOptions(token?: string): RequestInit { return token ? { headers: { Authorization: `Bearer ${token}` } } : {}; } diff --git a/src/github/parse.ts b/src/github/parse.ts index a09b110..a3c8179 100644 --- a/src/github/parse.ts +++ b/src/github/parse.ts @@ -1,3 +1,4 @@ +import { VALID_LOGIN } from "./constants.js"; import type { Source } from "./types.js"; export function parseSources(env: string | undefined): Source[] { @@ -33,3 +34,14 @@ export function parseNextLink(linkHeader: string | null): string | null { const match = linkHeader.match(/<([^>]+)>;\s*rel="next"/); return match?.[1] ?? null; } + +export function parseNames(raw: string, context: string): string[] { + const names: string[] = []; + const seen = new Set(); + for (const name of raw.split(',').map(s => s.trim()).filter(Boolean)) { + if (!VALID_LOGIN.test(name)) throw new Error(`${context}: "${name}" is not a valid GitHub account name`); + if (!seen.has(name.toLowerCase())) { seen.add(name.toLowerCase()); names.push(name); } + } + if (names.length === 0) throw new Error(`${context} contains no valid entries`); + return names; +} diff --git a/src/github/select.ts b/src/github/select.ts new file mode 100644 index 0000000..fee8afb --- /dev/null +++ b/src/github/select.ts @@ -0,0 +1,63 @@ +import { MAX_OPEN_SOURCES, VALID_LOGIN } from "./constants.js"; +import { parseNames } from "./parse.js"; + +export type Mode = + | { mode: "personal" } + | { mode: "enumerated"; allowed: string[] } + | { mode: "open" }; + +export class SelectionError extends Error {} + +export function detectMode(env: NodeJS.ProcessEnv): Mode { + const personal = !!(env["GITHUB_USERNAMES"]?.trim() || env["GITHUB_ORGS"]?.trim()); + const allowed = env["GITHUB_ALLOWED_SOURCES"]?.trim(); + + if (personal && allowed) throw new Error( + "GITHUB_ALLOWED_SOURCES cannot be combined with GITHUB_USERNAMES/GITHUB_ORGS" + ); + if (!personal && !allowed) throw new Error( + "Set GITHUB_USERNAMES/GITHUB_ORGS, or GITHUB_ALLOWED_SOURCES for a hosted instance" + ); + if (personal) return { mode: "personal" }; + if (allowed === "*") return { mode: "open" }; + return { mode: "enumerated", allowed: parseNames(allowed!, "GITHUB_ALLOWED_SOURCES") }; +} + +export function resolveSources(param: string | undefined, mode: Mode): string[] | null { + const names = param?.split(',').map(s => s.trim()).filter(Boolean) ?? []; + + if (names.length === 0) { + if (mode.mode === "open") throw new SelectionError("This instance requires ?source=[,...]"); + return null; + } + + if (mode.mode === "personal") throw new SelectionError("Source selection is not enabled on this instance"); + + if (mode.mode === "open") { + const unique = [...new Set(names.map(n => n.toLowerCase()))]; + if (unique.length > MAX_OPEN_SOURCES) throw new SelectionError( + `Too many sources: at most ${MAX_OPEN_SOURCES} per request` + ); + for (const name of unique) if (!VALID_LOGIN.test(name)) throw new SelectionError("Invalid source name"); + return unique; + } + + return [...new Set(names.map(n => { + const hit = mode.allowed.find(a => a.toLowerCase() === n.toLowerCase()); + if (!hit) throw new SelectionError("Unknown or disallowed source"); + return hit; + }))]; +} + +export function checkReferer(rawReferer: string | undefined, env: NodeJS.ProcessEnv): void { + const allowed = env["ALLOWED_REFERERS"]?.trim(); + if (!allowed) return; + + const hosts = allowed.split(',').map(s => s.trim().toLowerCase()).filter(Boolean); + let host: string | null = null; + try { host = rawReferer ? new URL(rawReferer).hostname.toLowerCase() : null; } catch { /* malformed header */ } + + if (!host || !hosts.includes(host)) throw new SelectionError( + "This instance only serves its own site - deploy your own from github.com/gh-top-languages/api" + ); +} diff --git a/src/github/types.ts b/src/github/types.ts index d1d89e3..10f65e5 100644 --- a/src/github/types.ts +++ b/src/github/types.ts @@ -1,3 +1,22 @@ +export type Repo = { + name: string; + fork: boolean; + full_name: string; +}; + export type Source = { name: string; token?: string }; +export type SourceKind = "user" | "org"; export type LanguageBytes = Record; + +export type FetchOutcome = + | { kind: "ok"; data: LanguageBytes } + | { kind: "missing" } + | { kind: "failed"; data: LanguageBytes; retryAt: number | null }; + +export type CacheEntry = { + data: LanguageBytes | null; + inFlight: Promise | null; + lastRefresh: number; + missingUntil?: number; +}; diff --git a/src/handler.ts b/src/handler.ts index ed42960..dbe215a 100644 --- a/src/handler.ts +++ b/src/handler.ts @@ -3,8 +3,9 @@ import { parseQueryParams, type QueryParams } from "@gh-top-languages/lib/utils/ import { generateChartData } from "@gh-top-languages/lib/charts/generate.js"; import { renderSvg } from "@gh-top-languages/lib/render/svg.js"; import { renderError } from "@gh-top-languages/lib/render/error.js"; -import { fetchLanguageData } from "./github/fetch.js"; -import { processLanguageData } from "./github/process.js"; +import { checkReferer, detectMode, resolveSources, SelectionError } from "./github/select.js"; +import { fetchLanguageData, fetchSelectedSources, SourceNotFoundError } from "./github/fetch.js"; +import { processLanguageData } from "./github/process.js"; export type ChartResponse = { status: number; @@ -17,12 +18,17 @@ export type RawQuery = Record; const first = (v: string | string[] | undefined): string | undefined => Array.isArray(v) ? v[0] : v; -export async function handleLanguages(rawQuery: RawQuery): Promise { +export async function handleLanguages( + rawQuery: RawQuery, + rawHeaders: RawQuery = {} +): Promise { const query: QueryParams = Object.fromEntries( Object.entries(rawQuery).map(([k, v]) => [k, first(v)]) ); try { + checkReferer(first(rawHeaders["origin"]) ?? first(rawHeaders["referer"]), process.env); + const { chartType, chartTitle, width, height, count, @@ -32,7 +38,15 @@ export async function handleLanguages(rawQuery: RawQuery): Promise { + const mode = detectMode(process.env); + const selected = resolveSources(query["source"], mode); + return selected ? fetchSelectedSources(selected, true ) + : mode.mode === "enumerated" ? fetchSelectedSources(mode.allowed, false) + : fetchLanguageData(); + })(); + const normalizedData = processLanguageData(rawData, count); const chart = generateChartData(normalizedData, selectedTheme, chartType, gapType, stroke); const svg = renderSvg(width, height, selectedTheme.bg, chart, chartTitle, selectedTheme.text); @@ -46,11 +60,12 @@ export async function handleLanguages(rawQuery: RawQuery): Promise embeds (camo proxy drops non-200 bodies) headers: { "Content-Type": "image/svg+xml", - "Cache-Control": "no-store", + "Cache-Control": requestShaped ? "public, max-age=300" : "no-store", "X-Chart-Error": "true" }, body: renderError((error as Error).message, DEFAULT_CONFIG.WIDTH, DEFAULT_CONFIG.HEIGHT) diff --git a/src/server.ts b/src/server.ts index d587e2a..22143d4 100644 --- a/src/server.ts +++ b/src/server.ts @@ -19,7 +19,7 @@ export function startServer(port: number): Server { return; } - const { status, headers, body } = await handleLanguages(queryFromUrl(url)); + const { status, headers, body } = await handleLanguages(queryFromUrl(url), req.headers as RawQuery); res.writeHead(status, headers); res.end(body); } catch { diff --git a/tests/api/languages/index.test.ts b/tests/api/languages/index.test.ts index ba19aa2..fbbb871 100644 --- a/tests/api/languages/index.test.ts +++ b/tests/api/languages/index.test.ts @@ -1,13 +1,13 @@ -import { describe, it, expect, vi, beforeEach } from "vitest"; -import type { VercelRequest, VercelResponse } from "@vercel/node"; -import { parseQueryParams } from "@gh-top-languages/lib/utils/params.js"; -import type { ChartResult } from "@gh-top-languages/lib/charts/types.js"; -import { generateChartData } from "@gh-top-languages/lib/charts/generate.js"; -import { renderSvg } from "@gh-top-languages/lib/render/svg.js"; -import { renderError } from "@gh-top-languages/lib/render/error.js"; -import handler from "../../../api/languages/index.js"; -import { fetchLanguageData } from "../../../src/github/fetch.js"; -import { processLanguageData } from "../../../src/github/process.js"; +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import type { VercelRequest, VercelResponse } from "@vercel/node"; +import { parseQueryParams } from "@gh-top-languages/lib/utils/params.js"; +import type { ChartResult } from "@gh-top-languages/lib/charts/types.js"; +import { generateChartData } from "@gh-top-languages/lib/charts/generate.js"; +import { renderSvg } from "@gh-top-languages/lib/render/svg.js"; +import { renderError } from "@gh-top-languages/lib/render/error.js"; +import handler from "../../../api/languages/index.js"; +import { fetchLanguageData } from "../../../src/github/fetch.js"; +import { processLanguageData } from "../../../src/github/process.js"; vi.mock("@gh-top-languages/lib/utils/params.js"); vi.mock("@gh-top-languages/lib/charts/generate.js"); @@ -28,6 +28,7 @@ describe("handler", () => { } as const; beforeEach(() => { + vi.stubEnv("GITHUB_USERNAMES", "testuser"); vi.clearAllMocks(); req = { query: {} } as VercelRequest; res = { @@ -50,6 +51,10 @@ describe("handler", () => { }); }); + afterEach(() => { + vi.unstubAllEnvs(); + }); + it("renders SVG successfully with valid data", async () => { const rawData = { JavaScript: 5000, Python: 3000 }; const normalizedData = [ @@ -70,7 +75,7 @@ describe("handler", () => { await handler(req, res); - expect(fetchLanguageData).toHaveBeenCalledWith(false); + expect(fetchLanguageData).toHaveBeenCalledWith(); expect(processLanguageData).toHaveBeenCalledWith(rawData, 5); expect(generateChartData).toHaveBeenCalledWith(normalizedData, mockTheme, "donut", "gap", false); expect(renderSvg).toHaveBeenCalledWith(600, 400, "#ffffff", chartData, "Languages", "#000000"); diff --git a/tests/github/fetch.test.ts b/tests/github/fetch.test.ts index 8abaf36..7f4cd51 100644 --- a/tests/github/fetch.test.ts +++ b/tests/github/fetch.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; -import { fetchLanguageData, resetCache } from "../../src/github/fetch.js"; +import { fetchLanguageData, fetchSelectedSources, resetCache } from "../../src/github/fetch.js"; const repos = [ { name: "repo1", fork: false, full_name: "user/repo1" }, @@ -155,7 +155,7 @@ describe("fetchLanguageData", () => { const result1 = await fetchLanguageData(); const result2 = await fetchLanguageData(); - expect(result1).toBe(result2); + expect(result1).toStrictEqual(result2); expect(global.fetch).toHaveBeenCalledTimes(2); }); @@ -170,7 +170,7 @@ describe("fetchLanguageData", () => { ]); expect(global.fetch).toHaveBeenCalledTimes(2); - expect(result1).toBe(result2); + expect(result1).toStrictEqual(result2); }); it("handles failed language fetch gracefully", async () => { @@ -516,4 +516,53 @@ describe("fetchLanguageData", () => { "https://api.github.com/users/testuser/repos?per_page=100", {} ); }); + + it("strict and non-strict callers racing on a 404 each get their own semantics", async () => { + mockFetch().mockResolvedValue(mockErrorResponse(404, "Not Found")); + + const strictP = fetchSelectedSources(["ghost"], true); + const nonStrictP = fetchSelectedSources(["ghost"], false); + + await expect(strictP).rejects.toThrow("Unknown GitHub account"); + await expect(nonStrictP).resolves.toEqual({}); + expect(global.fetch).toHaveBeenCalledTimes(1); + }); + + it("strict caller hitting the negative cache is rejected without a fetch", async () => { + mockFetch().mockResolvedValue(mockErrorResponse(404, "Not Found")); + await fetchSelectedSources(["ghost"], false); + + mockFetch().mockClear(); + await expect(fetchSelectedSources(["ghost"], false)).resolves.toEqual({}); + await expect(fetchSelectedSources(["ghost"], true)).rejects.toThrow("Unknown GitHub account"); + expect(global.fetch).not.toHaveBeenCalled(); + }); + + it("strict caller gets an error on an uncached transient failure; non-strict gets empty data", async () => { + mockFetch().mockResolvedValue(mockErrorResponse(500, "Internal Server Error")); + + await expect(fetchSelectedSources(["someuser"], true)).rejects.toThrow("GitHub API error: source fetch failed"); + resetCache(); + mockFetch().mockClear().mockResolvedValue(mockErrorResponse(500, "Internal Server Error")); + await expect(fetchSelectedSources(["someuser"], false)).resolves.toEqual({}); + }); + + it("dedupes case-variant configured sources so bytes are counted once", async () => { + vi.stubEnv("GITHUB_USERNAMES", `["testuser", "TESTUSER"]`); + + mockFetch() + .mockResolvedValueOnce(mockResponse(repos)) + .mockResolvedValueOnce(mockResponse(languages)); + + const result = await fetchLanguageData(); + expect(global.fetch).toHaveBeenCalledTimes(2); + expect(result).toEqual({ JavaScript: 5000, Python: 3000, HTML: 2000 }); + }); + + it("hosted fetches use the public users endpoint with no per-source auth", async () => { + vi.stubEnv("GITHUB_TOKEN", ""); + mockFetch().mockResolvedValueOnce(mockResponse([])); + await expect(fetchSelectedSources(["someuser"], true)).resolves.toEqual({}); + expect(global.fetch).toHaveBeenCalledWith("https://api.github.com/users/someuser/repos?per_page=100", {}); + }); }); diff --git a/tests/github/select.test.ts b/tests/github/select.test.ts new file mode 100644 index 0000000..d56ea45 --- /dev/null +++ b/tests/github/select.test.ts @@ -0,0 +1,64 @@ +import { describe, it, expect } from "vitest"; +import { detectMode, resolveSources, type Mode } from "../../src/github/select.js"; + +const personal: Mode = { mode: "personal" }; +const open: Mode = { mode: "open" }; +const enumerated: Mode = { mode: "enumerated", allowed: ["Mason", "acme"] }; + +describe("detectMode", () => { + it("personal via GITHUB_USERNAMES", () => + expect(detectMode({ GITHUB_USERNAMES: "a" })).toEqual(personal)); + it("personal via GITHUB_ORGS", () => + expect(detectMode({ GITHUB_ORGS: "a" })).toEqual(personal)); + it("throws when both consumer vars set", () => + expect(() => detectMode({ GITHUB_USERNAMES: "a", GITHUB_ALLOWED_SOURCES: "b" })) + .toThrow(/cannot be combined/)); + it("throws when neither is set", () => + expect(() => detectMode({})).toThrow(/Set GITHUB_USERNAMES/)); + it("treats whitespace-only vars as unset",() => expect(() => + detectMode({ GITHUB_USERNAMES: " " })).toThrow(/Set GITHUB_USERNAMES/)); + it("* means open mode", () => + expect(detectMode({ GITHUB_ALLOWED_SOURCES: "*" })).toEqual(open)); + it("names mean enumerated mode", () => + expect(detectMode({ GITHUB_ALLOWED_SOURCES: "Mason, acme" })).toEqual(enumerated)); + it("throws on invalid name in allowlist", () => + expect(() => detectMode({ GITHUB_ALLOWED_SOURCES: "ok,-bad-" })).toThrow(/not a valid/)); + it("throws on comma-only allowlist", () => + expect(() => detectMode({ GITHUB_ALLOWED_SOURCES: ",," })).toThrow(/no valid entries/)); + it("dedupes case-variant allowlist entries", () => + expect(detectMode({ GITHUB_ALLOWED_SOURCES: "mason,MASON" })).toEqual({ mode: "enumerated", allowed: ["mason"] })); +}); + +describe("resolveSources", () => { + it("absent param → null in personal", () => + expect(resolveSources(undefined, personal)).toBeNull()); + it("absent param → null in enumerated", () => + expect(resolveSources(undefined, enumerated)).toBeNull()); + it("empty/degenerate param behaves as absent", () => { + expect(resolveSources("", enumerated)).toBeNull(); + expect(resolveSources(" ,, ", enumerated)).toBeNull(); + }); + it("absent param → throws in open", () => + expect(() => resolveSources(undefined, open)).toThrow(/requires \?source=/)); + it("any param → throws in personal", () => + expect(() => resolveSources("mason", personal)).toThrow(/not enabled/)); + + it("enumerated match is case-insensitive, returns canonical casing", + () => expect(resolveSources("MASON,ACME", enumerated)).toEqual(["Mason", "acme"])); + it("enumerated unknown name → throws", () => + expect(() => resolveSources("stranger", enumerated)).toThrow(/Unknown or disallowed/)); + it("one bad among good fails the whole request", + () => expect(() => resolveSources("Mason,stranger", enumerated)).toThrow(/Unknown or disallowed/)); + + it("enumerated mode dedupes to one canonical entry", () => expect(resolveSources("Mason,MASON", enumerated)).toEqual(["Mason"])); + it("open mode dedupes case-variant names", () => expect(resolveSources("Mason,mason,MASON", open)).toEqual(["mason"])); + it("open passes valid names through", () => + expect(resolveSources("a,b-c", open)).toEqual(["a", "b-c"])); + it("open rejects more than 10 names", () => + expect(() => resolveSources("a,b,c,d,e,f,g,h,i,j,k", open)).toThrow(/Too many/)); + it("open rejects malformed logins", () => { + expect(() => resolveSources("-bad-", open)).toThrow(/Invalid source name/); + expect(() => resolveSources("a".repeat(40), open)).toThrow(/Invalid source name/); + expect(() => resolveSources("double--hyphen", open)).toThrow(/Invalid source name/); + }); +}); diff --git a/tests/handler.test.ts b/tests/handler.test.ts index 9281727..8667b54 100644 --- a/tests/handler.test.ts +++ b/tests/handler.test.ts @@ -1,5 +1,6 @@ -import { describe, it, expect } from "vitest"; -import { handleLanguages } from "../src/handler.js"; +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { handleLanguages } from "../src/handler.js"; +import { resetCache } from "../src/github/fetch.js"; describe("handleLanguages", () => { it("takes the first value when a query param is supplied as an array", async () => { @@ -8,3 +9,139 @@ describe("handleLanguages", () => { expect(res.body).toContain('fill="#ff0000"'); }); }); + +describe("source selection", () => { + const mockGitHub = (sources: Record | number>) => { + global.fetch = vi.fn(async (url: RequestInfo | URL) => { + const u = String(url); + for (const [name, langs] of Object.entries(sources)) { + if (u.includes(`/users/${name}/repos`)) { + if (typeof langs === "number") return { ok: false, status: langs, statusText: "", headers: { get: () => null } } as unknown as Response; + return { ok: true, json: async () => [{ name: `${name}-repo`, fork: false, full_name: `${name}/${name}-repo` }], headers: { get: () => null } } as unknown as Response; + } + if (u.includes(`/repos/${name}/`)) return { ok: true, json: async () => langs, headers: { get: () => null } } as unknown as Response; + } + throw new Error(`unexpected fetch: ${u}`); + }) as typeof fetch; + }; + + beforeEach(() => { + resetCache(); + vi.spyOn(console, "error").mockImplementation(() => {}); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); + }); + + it("personal mode rejects ?source=", async () => { + vi.stubEnv("GITHUB_USERNAMES", "me"); + const res = await handleLanguages({ source: "other" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + }); + + it("enumerated bare URL renders the aggregate", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "alice,bob"); + mockGitHub({ alice: { Rust: 9000 }, bob: { Go: 8000 } }); + const res = await handleLanguages({}); + expect(res.headers["X-Chart-Error"]).toBeUndefined(); + expect(res.body).toContain("Rust"); + expect(res.body).toContain("Go"); + }); + + it("enumerated ?source= renders only the selected source", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "alice,bob"); + mockGitHub({ alice: { Rust: 9000 }, bob: { Go: 8000 } }); + const res = await handleLanguages({ source: "alice" }); + expect(res.body).toContain("Rust"); + expect(res.body).not.toContain("Go"); + }); + + it("enumerated unknown source renders a cacheable error SVG", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "alice"); + const res = await handleLanguages({ source: "stranger" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + expect(res.body).not.toContain("stranger"); + }); + + it("open bare URL renders a cacheable error SVG", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "*"); + const res = await handleLanguages({}); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + }); + + it("open ?source= hitting a GitHub 404 renders a cacheable error SVG", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "*"); + mockGitHub({ ghost: 404 }); + const res = await handleLanguages({ source: "ghost" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + }); + + it("open mode rejects more than 10 sources", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "*"); + const res = await handleLanguages({ source: "a,b,c,d,e,f,g,h,i,j,k" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + }); + + it("both consumer vars set is a server-side error, not cacheable", async () => { + vi.stubEnv("GITHUB_USERNAMES", "me"); + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "alice"); + const res = await handleLanguages({}); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("no-store"); + }); + + it("duplicate names in ?source= are counted once", async () => { + vi.stubEnv("GITHUB_ALLOWED_SOURCES", "*"); + mockGitHub({ alice: { Rust: 9000 } }); + const res = await handleLanguages({ source: "alice,alice" }); + expect(res.headers["X-Chart-Error"]).toBeUndefined(); + expect(global.fetch).toHaveBeenCalledTimes(2); + }); +}); + +describe("referer gate", () => { + afterEach(() => vi.unstubAllEnvs()); + + it("unset ALLOWED_REFERERS leaves the instance open", async () => { + const res = await handleLanguages({ test: "true" }); + expect(res.headers["X-Chart-Error"]).toBeUndefined(); + }); + + it("matching origin passes", async () => { + vi.stubEnv("ALLOWED_REFERERS", "builder.example.com"); + const res = await handleLanguages({ test: "true" }, { origin: "https://builder.example.com" }); + expect(res.headers["X-Chart-Error"]).toBeUndefined(); + }); + + it("matching referer passes when origin is absent", async () => { + vi.stubEnv("ALLOWED_REFERERS", "builder.example.com"); + const res = await handleLanguages({ test: "true" }, { referer: "https://builder.example.com/builder?theme=dark" }); + expect(res.headers["X-Chart-Error"]).toBeUndefined(); + }); + + it("foreign referer is blocked with a cacheable error", async () => { + vi.stubEnv("ALLOWED_REFERERS", "builder.example.com"); + const res = await handleLanguages({ test: "true" }, { referer: "https://freeloader.example.com/" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + expect(res.headers["Cache-Control"]).toBe("public, max-age=300"); + }); + + it("missing referer is blocked (camo/hotlink case)", async () => { + vi.stubEnv("ALLOWED_REFERERS", "builder.example.com"); + const res = await handleLanguages({ test: "true" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + }); + + it("substring lookalike host is blocked", async () => { + vi.stubEnv("ALLOWED_REFERERS", "builder.example.com"); + const res = await handleLanguages({ test: "true" }, { referer: "https://builder.example.com.evil.com/" }); + expect(res.headers["X-Chart-Error"]).toBe("true"); + }); +});