Skip to content

Add 5 journey skills: fundamentals, agent-infrastructure, secure-delivery, build-investigation, hosted-agents-operations#32

Draft
simone-bennett wants to merge 1 commit into
mainfrom
add-buildkite-journey-skills
Draft

Add 5 journey skills: fundamentals, agent-infrastructure, secure-delivery, build-investigation, hosted-agents-operations#32
simone-bennett wants to merge 1 commit into
mainfrom
add-buildkite-journey-skills

Conversation

@simone-bennett

Copy link
Copy Markdown
Contributor

Summary

Adds five new Buildkite skills following the Honeycomb-inspired "small + opinionated" shape — one anchor skill + three setup/journey skills + one investigation skill. Each is grounded in real customer escalations from Linear, with cross-source signal triangulation from docs / MRR pain CSVs / external benchmarking.

The shape mirrors what Honeycomb shipped with their agent-skill repo — investigation loop, framework-specific gotcha tables, opinionated key principles — adapted to Buildkite's product surface.

What's added

Skill Body Purpose Anchored in
buildkite-fundamentals ~1.9K w Anchor: build hierarchy, agent/queue/cluster, MCP vs REST vs CLI surface map Structural
buildkite-agent-infrastructure ~3.5K w 7-phase setup: platform choice, queue design, k8s controller, hooks, tokens, observability, EC2→K8s migration PS-1200, PS-1250, A-1110, PS-1000, PS-1800 (Mistral, Doordash, Dropbox, Equilibrium, Meta)
buildkite-secure-delivery ~3.3K w OIDC, secrets workflow, signed pipelines, token lifecycle, Package Registries auth, SLSA mapping, untrusted-data handling OIDC docs fragmentation; 5 SSO/token pain customers; verbatim doc quotes
buildkite-build-investigation ~3.1K w Named workflow: Orient → Localize → Correlate → Verify → Record. Plus ## Stay on the Path anti-bias section and 20+ row failure-attribution decision tree PS-555 (8-customer cohort), PS-1300 (Spotify), PS-1600 (Anthropic Sev 5), PS-500 (Boston Dynamics), A-1110, PF-9400
buildkite-hosted-agents-operations ~3.1K w Image lifecycle, macOS/Xcode/fastlane, cache volumes, namespace concurrency, terminal access A-500 (Groq image push), PS-400 (11x.ai namespace), 7 Mac/iOS pain customers

Each skill includes:

  • SKILL.md (mandatory section order)
  • agents/openai.yaml with Buildkite brand color #00D974
  • assets/.gitkeep (icon PNGs TBD via brand team)
  • references/ (4-6 files per skill)
  • scripts/ where helpful (5 scripts total: agent-stack-k8s-diagnose, investigate-build, check-pipeline-for-secrets, oidc-subject-claim-preview, hosted-image-build-and-push)
  • evals/evals.md (markdown table format; eval YAML for the internal harness lands in skills-internal-tools PR)

CONVENTIONS.md / AGENTS.md updates

  • Boundary table: adds 5 new skill rows; tightens buildkite-agent-infrastructure scope to what the skill actually covers; drops SSO/SAML, audit logging, pipeline templates, cost optimization (documented as gaps for future work — see "Topics deferred from this batch" note in CONVENTIONS.md)
  • Section order: adds optional ## Anti-Scope section between Additional Resources and Further Reading
  • Quick Start: now permits a small decision table or symptom router preceding the < 20 line code block (used by agent-infrastructure and hosted-agents-operations)
  • Size targets: body raised from 10-18 KB to 10-25 KB; total from 15-45 KB to 15-100 KB (matches actual density of journey skills with verbatim doc quotes and gotcha tables; the user-edited buildkite-pipelines/SKILL.md is already at ~22 KB)
  • Repository Structure in AGENTS.md: adds the 5 new skill names + restores explicit listing of buildkite-preflight

Verification

  • ./scripts/build-power.sh runs cleanly; steering/ regenerated (CI drift check should pass)
  • Every cross-reference points to a real skill (no (planned) qualifiers on in-batch siblings)
  • Section order compliant across all 5 (Quick Start immediately after overview; Common Mistakes → Additional Resources → Anti-Scope → Further Reading)
  • All scripts: shebang, set -euo pipefail, header docs, bash -n parses, executable
  • Cross-source verification: review v1 + v2 against draft state preserved in skills-research/04-review.md and 04-review-v2.md (kept private)

Engineer verification needed before merge

These items I could not verify from research summaries alone — please sanity-check before approval:

  1. skills/buildkite-agent-infrastructure/SKILL.md Phase 3 gotcha table — exact agent + controller version pins (Mistral localHookPath regression, Doordash viper hugepages version, chart v0.30.x build-dir change)
  2. skills/buildkite-hosted-agents-operations/references/macos-build-gotchas.md — current Xcode / macOS version matrix
  3. skills/buildkite-secure-delivery/references/slsa-mapping.mdbuildType: "https://buildkite.com/build-types/v1" URL (verify Buildkite has published this identifier)
  4. skills/buildkite-agent-infrastructure/SKILL.md Quick Start — Elastic CI Stack CloudFormation template URL (https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml)

Out of scope for this PR

  • Icon PNGs for the 5 new assets/ directories (placeholder .gitkeep for now)
  • The skills-internal-tools eval dataset entries (separate PR there)
  • A future buildkite-admin skill covering SSO/SAML, audit logging, pipeline templates, cost optimization

Test plan

  • CI drift check passes (steering/ in sync with skills/)
  • Engineer review confirms version pins / Xcode matrix / SLSA buildType / CFN URL
  • Internal eval harness pass (separate PR in buildkite/skills-internal-tools)
  • Brand team adds icon PNGs to assets/ directories

Adds:
- buildkite-fundamentals     — anchor skill: core vocabulary, build hierarchy, surface map
- buildkite-agent-infrastructure — self-hosted clusters, K8s, queue design, lifecycle hooks, tokens
- buildkite-secure-delivery   — OIDC, secrets workflows, signed pipelines, SLSA, Package Registries auth
- buildkite-build-investigation — Orient/Localize/Correlate/Verify/Record investigation workflow
- buildkite-hosted-agents-operations — Buildkite-hosted Linux + macOS runbook (image lifecycle, Xcode, namespace)

Each skill follows the canonical SKILL.md format and is grounded in real customer
escalations (Linear PS-*/A-*/PF-* ticket IDs cited verbatim in references and evals).
All five pass through scripts/build-power.sh cleanly; steering/ is regenerated.

CONVENTIONS.md changes:
- Boundary table: adds the 5 new skill rows, tightens agent-infrastructure scope to
  what the skill actually covers, drops deferred topics (SSO/SAML, audit logging,
  pipeline templates, cost optimization) — documented as gaps for future work
- Section order: adds optional `## Anti-Scope` between Additional Resources and Further Reading
- Quick Start: permits a small decision table or symptom router preceding the code block
- Size targets: raises body cap to 25KB and total to 100KB (matches journey-skill density)

AGENTS.md changes:
- Repository Structure list: adds the 5 new skills
- Size targets updated to match CONVENTIONS.md

Engineer verification still needed before merge:
- agent-infrastructure gotcha-table: exact agent/controller version pins
- hosted-agents Xcode/macOS matrix in references/macos-build-gotchas.md
- secure-delivery SLSA buildType URL in references/slsa-mapping.md
- agent-infrastructure Quick Start: Elastic CI Stack CloudFormation template URL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant