From 0f969e2b70d2603f572827562618e791a690f040 Mon Sep 17 00:00:00 2001 From: fredbi Date: Mon, 6 Jul 2026 22:25:16 +0200 Subject: [PATCH 1/2] doc(skills): monthly-newsletter data collection via git, not the REST API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scheduled cloud routine's egress is bound to a single repo, so every api.github.com / github.com / codeload request returns 403 — including unauthenticated public reads and the org-listing endpoints gh repo list uses. Rewrite the data-collection method to use the git smart-HTTP protocol (clone, ls-remote) and raw.githubusercontent.com, which the proxy does allow: - enumerate repos from go.mod unions + known infra, validated with git ls-remote - collect commits/releases from shallow --shallow-since clones and tags - recover contributor handles from noreply emails and commit trailers (no .author.login) - document the MCP Contents-API publish path (no git push: unsigned/proxy-attributed) Signed-off-by: Frédéric BIDON Co-Authored-By: Claude --- .claude/skills/monthly-newsletter.md | 124 +++++++++++++++++---------- 1 file changed, 79 insertions(+), 45 deletions(-) diff --git a/.claude/skills/monthly-newsletter.md b/.claude/skills/monthly-newsletter.md index 158436d..9d1c3ed 100644 --- a/.claude/skills/monthly-newsletter.md +++ b/.claude/skills/monthly-newsletter.md @@ -106,56 +106,79 @@ Close with a short, warm thank-you to the **external human contributors** for th ## Data collection (cloud environment) -The routine starts with **only `doc-site` checked out**, so prefer the **GitHub API** -over cloning ~18 repos. **All reads must be authenticated** — unauthenticated GitHub -API is 60 req/hour (far too low); authenticated is 5,000 req/hour. The `gh` CLI uses -the `GH_TOKEN` env var automatically. - -1. **Enumerate repositories** in both orgs (skip archived repos and forks): - ```bash - gh repo list go-openapi --no-archived --source --limit 100 --json name,isArchived,isFork \ - --jq '.[] | select(.isArchived|not) | select(.isFork|not) | .name' - gh repo list go-swagger --no-archived --source --limit 100 --json name,isArchived,isFork \ - --jq '.[] | select(.isArchived|not) | select(.isFork|not) | .name' - ``` - -2. **Per repo, over the window**, pull what you need via the API (replace `/`, - ``, ``): - - Commits on the default branch: +The routine starts with **only `doc-site` checked out**. **Do not** try to use the +GitHub REST API (`gh api`, `curl https://api.github.com/...`) or `gh repo list`: in the +scheduled cloud sandbox the egress proxy binds the session to a single repository, so +**every `api.github.com`, `github.com`, and `codeload.github.com` request returns HTTP +403** — including *unauthenticated public reads* and the org-listing / GraphQL endpoints +`gh repo list` depends on. A token does not help; the block is at the network layer. + +What the proxy **does** allow, and what this routine relies on: + +- the **git smart-HTTP protocol** — `git clone`, `git ls-remote`, `git fetch` — against + any public repo (and the `fredbi/doc-site` fork), and +- **`raw.githubusercontent.com`** (single-file reads). + +Public repos need **no token** for git reads. So collect everything from **shallow git +clones** rather than the API. Work in a scratch dir, not the `doc-site` checkout. + +1. **Enumerate repositories.** The org-listing API is blocked, so build the repo set from + readable sources and validate it with `git ls-remote` (skip archived repos and forks — + forks are absent from these orgs in practice; dormant repos fall out naturally in step 2): + - **Libraries** — union of `github.com/go-openapi/*` module paths found in a few + `go.mod` files (fetch via raw; go-swagger's `go.mod` pulls in most of them): ```bash - gh api -X GET "repos///commits" \ - -f since= -f until= --paginate \ - --jq '.[] | {sha:.sha, msg:(.commit.message|split("\n")[0]), login:.author.login, name:.commit.author.name}' + for m in go-swagger/go-swagger go-openapi/runtime go-openapi/validate go-openapi/strfmt; do + curl -sS "https://raw.githubusercontent.com/$m/master/go.mod" + done | grep -oE 'go-openapi/[a-z0-9-]+' | sort -u ``` - - Releases published in the window: + - **Non-module repos** — add the known infra/tooling/example repos not referenced by any + `go.mod`: `go-openapi/{doc-site,ci-workflows,.github,codescan,kvstore,stubs,swaggersocket}` + and `go-swagger/{go-swagger,examples,scan-repo-boundary}`. + - **Validate** each candidate exists and is reachable (drop the misses): ```bash - gh api "repos///releases" --paginate \ - --jq '.[] | {tag:.tag_name, published:.published_at}' + git ls-remote "https://github.com//" HEAD >/dev/null 2>&1 && echo "/" ``` - - Latest release tag (for the highlights table). **Do not** use `releases/latest` — - it returns HTTP 404 for repos with no published release and `gh` leaks the error - body to stdout. Ask for the newest release (or newest tag) instead, which yields an - empty array, not a 404: + +2. **Per repo, clone shallow over the window** (start a few days *before* `` so the + first in-window commit is included; `--no-checkout --filter=blob:none` keeps it cheap — + we only need history, not trees): + ```bash + git clone -q --no-checkout --filter=blob:none --shallow-since= \ + "https://github.com//" "" + ``` + - **Commits in the window** (author name, email, subject): ```bash - tag=$(gh api "repos///releases?per_page=1" --jq '.[0].tag_name // empty' 2>/dev/null) - [ -z "$tag" ] && tag=$(gh api "repos///tags?per_page=1" --jq '.[0].name // empty' 2>/dev/null) - # repos with neither (e.g. codegen, doc-site) → leave the cell as "—" + git -C log --since= --until= --pretty='%an%x09%ae%x09%s' ``` - - Merged PRs in the window (optional, for richer highlights): + A repo whose newest commit predates `--shallow-since` **fails to clone** — that just + means zero activity in the window. Confirm dormancy if unsure with a + `git clone --depth=1` and `git -C log -1 --pretty=%ci`. + - **Releases in the window** and **latest release tag** (for the highlights table) from + tags — no Releases API needed: ```bash - gh api -X GET "search/issues" \ - -f q='repo:/ is:pr is:merged merged:..' \ - --jq '.items[] | {n:.number, title:.title, user:.user.login}' + # tags whose commit date lands in the window: + git -C for-each-ref --format='%(refname:short) %(creatordate:short)' refs/tags \ + | awk '$2>="" && $2<""' + # newest semver release tag (ignores per-submodule path tags): + git -C ls-remote --tags origin | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1 ``` - -3. **Shallow-clone only if needed** for file-level inspection of one repo: - `git clone --depth=50 --shallow-since= https://github.com//`. - -4. **Contributor handles** come straight from the commit API (`.author.login` above); - no SHA-to-login mapping needed. Skip null logins (web-flow / unmatched) rather than - guessing. - -5. **Cap and log truncation.** If any repo's history is too large to page fully within + Repos with no version tag (e.g. `doc-site`, `.github`) → leave the cell as "—". + - **Merged-PR numbers** (optional, for richer highlights) are embedded in squashed commit + subjects as `(#NNN)` — extract them from the `%s` output rather than the search API. + +3. **Contributor handles without the API.** Git carries author *name + email*, not the + GitHub login, so recover the handle from what git *does* have — never guess, never list + an email: + - **GitHub noreply emails** encode the login: `NNN+@users.noreply.github.com` + → ``. + - **`Signed-off-by:` / `Co-authored-by:` trailers** in the full commit body + (`git log --pretty='%an%x09%s%x09%b'`) frequently carry the handle. + - If a contributor's handle can't be derived from either (author used a plain email and + no trailer names them), **omit them** from the thanks section rather than guessing or + printing a display name — and `log` the omission so the gap is visible. + +4. **Cap and log truncation.** If any repo's history is too large to page fully within reason, `log` what was capped rather than silently truncating — a report that hides gaps reads as complete when it is not. @@ -172,6 +195,17 @@ to the user. ## After writing Write the file to `docs/doc-site/blog/monthly/-.md`. Do **not** post to -Discord and do **not** commit from the skill — the routine handles the commit (via the -GitHub Contents API) and the PR; the Discord announcement happens on merge via -`announce-monthly.yml`. +Discord and do **not** commit from the skill — the routine handles the commit and the PR; +the Discord announcement happens on merge via `announce-monthly.yml`. + +**Publishing note (same sandbox constraint).** Because `api.github.com` is blocked, the +routine's original `gh api .../contents/...` Contents-API path to the `fredbi/doc-site` +fork does **not** work, and a plain `git push` from the sandbox produces an *unsigned* +commit attributed to the proxy identity — which the contribution rules forbid. The +working path is the **in-scope GitHub MCP server** (`go-openapi/doc-site` only), which +reaches the Contents API server-side and yields a GitHub-Verified, token-owner-authored +commit: create a `claude/monthly-` branch, `create_or_update_file` the report onto it +(sign the message off as the maintainer), then open the PR on `go-openapi/doc-site` with +`base=master head=claude/monthly-`. This is a same-repo branch PR, not a cross-fork +one. Record any such deviation (branch-on-upstream vs fork, git-derived data vs API) in +the PR body so reviewers know how the run was produced. From 1cb8a077be44fc554286efb1d94735ee11a624a1 Mon Sep 17 00:00:00 2001 From: fredbi Date: Mon, 6 Jul 2026 22:39:07 +0200 Subject: [PATCH 2/2] doc(skills): let load-bearing commit volume drive the lede MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-mortem fix for the June run, which led with a "security" theme (a few commits) and filtered away the month's actual centre of gravity: 367 codescan commits (61% of the month) sweeping the go-swagger issue backlog into tests after the codescan extraction. - New "Finding the month's story" section: rank repos by load-bearing commit share; the top repo(s) MUST be characterized; a repo >~15-20% of commits may never be dismissed as noise even when chore:/test:-prefixed; investigate dominant chore:/test: clusters (their (#NNN) refs reveal backlog sweeps) instead of filtering them; connect the causal chain and get the timing right. - Data collection: exclude non-load-bearing bot authors (dependabot, bot-go-openapi/go-openapi-bot, github-actions) and automated contributors/prepare-release commits from the counts that drive the narrative. Signed-off-by: Frédéric BIDON Co-Authored-By: Claude --- .claude/skills/monthly-newsletter.md | 49 ++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/.claude/skills/monthly-newsletter.md b/.claude/skills/monthly-newsletter.md index 9d1c3ed..66ecb15 100644 --- a/.claude/skills/monthly-newsletter.md +++ b/.claude/skills/monthly-newsletter.md @@ -62,7 +62,10 @@ discord_description: |- A single short paragraph framing the month: the overall shape of activity and the one or two things worth noticing. No grand strategy, no marketing language. State the -**overall effort** inline (e.g. "N commits across M repositories"). +**overall effort** inline (e.g. "N commits across M repositories") — where **N is the +load-bearing commit count**, i.e. excluding the automated bot authors (see *Finding the +month's story* and *Data collection*). The lede must name whatever actually dominated +the month's work, not whatever reads most excitingly. ### 2. Themes (compact list) @@ -70,6 +73,7 @@ A short bulleted list of the month's cross-cutting themes — group similar chan across repos rather than listing per-repo. Typical buckets: features, bug fixes, dependency/CI maintenance, docs, releases. **Keep it to the few themes that actually mattered this month**; do not pad with boilerplate categories that saw no real change. +The theme carrying the largest share of load-bearing work leads the list. ### 3. Repository highlights (short table) @@ -104,6 +108,32 @@ Close with a short, warm thank-you to the **external human contributors** for th - If there were no external contributors this month, omit the section rather than writing an empty thank-you. +## Finding the month's story (before you write) + +The failure mode this section exists to prevent: keyword-hunting for an attractive theme +(a GHSA fix, a "security hardening" label) and making it the lede, while the activity that +actually consumed the month gets filtered away as `chore:` noise. **Let the volume of +load-bearing work decide the lede, then explain it.** A vivid handful of commits is not the +headline just because it reads like one. + +1. **Rank repos by load-bearing commit share.** Using the bot-filtered counts (see Data + collection), compute each repo's share of the month's total commits. +2. **The top one or two repos MUST be characterized** in the intro and themes. Any repo + holding **more than ~15–20%** of the month's load-bearing commits may **never** be + omitted or waved off as noise — *even if* its commits are `chore:`/`test:`-prefixed. +3. **A dominant `chore:`/`test:` cluster is a signal to investigate, not filter.** Read the + actual subjects: they often carry `(#NNN)` / `(go-swagger#NNN)` issue references that + reveal the real story — a backlog sweep, a test-migration, an issue-triage pass. Do not + let the noise filter you use for the highlights table hide the month's biggest theme. +4. **Connect the causal chain, and get the timing right.** A repo move, a major release, and + a wave of follow-on commits in *another* repo are usually one story, not three separate + bullets. Check when each cause actually happened — the cause may predate the window while + its consequence fills it (e.g. a repo extraction one month, the dependent cleanup the + next) — and state the sequence rather than mis-dating it. +5. **Sanity check before writing.** Does your framing explain where the commits actually + went? If the themes describe 5% of the month and ignore the 60%, the framing is wrong — + redo it. + ## Data collection (cloud environment) The routine starts with **only `doc-site` checked out**. **Do not** try to use the @@ -154,6 +184,19 @@ clones** rather than the API. Work in a scratch dir, not the `doc-site` checkout A repo whose newest commit predates `--shallow-since` **fails to clone** — that just means zero activity in the window. Confirm dormancy if unsure with a `git clone --depth=1` and `git -C log -1 --pretty=%ci`. + - **Load-bearing commit count.** Every count you report or rank on — the intro's "N + commits" and the *Finding the month's story* ranking — must **exclude non-load-bearing + automated authors**, which inflate totals and never carry a theme: + `dependabot[bot]`, `bot-go-openapi[bot]` / `go-openapi-bot`, `github-actions[bot]`, + plus the automated `doc: updated contributors file` and `chore: prepare release …` + commits. Filter on author/email (and drop those two subjects): + ```bash + git -C log --since= --until= --pretty='%an%x09%ae%x09%s' \ + | grep -vaiE 'dependabot|bot-go-openapi|go-openapi-bot|github-actions' \ + | grep -vaiE 'updated contributors file|prepare release' + ``` + Keep the raw count too if you like, but the **load-bearing** count is the one that + drives the narrative and the reported effort figure. - **Releases in the window** and **latest release tag** (for the highlights table) from tags — no Releases API needed: ```bash @@ -165,7 +208,9 @@ clones** rather than the API. Work in a scratch dir, not the `doc-site` checkout ``` Repos with no version tag (e.g. `doc-site`, `.github`) → leave the cell as "—". - **Merged-PR numbers** (optional, for richer highlights) are embedded in squashed commit - subjects as `(#NNN)` — extract them from the `%s` output rather than the search API. + subjects as `(#NNN)` / `(go-swagger#NNN)` — extract them from the `%s` output rather + than the search API. These refs are also the thread that reveals backlog sweeps and + issue-triage passes (see *Finding the month's story*). 3. **Contributor handles without the API.** Git carries author *name + email*, not the GitHub login, so recover the handle from what git *does* have — never guess, never list