Skip to content

refactor: replace protobuf submodule with vcpkg#162

Merged
wenchy merged 67 commits into
masterfrom
vcpkg
Jun 9, 2026
Merged

refactor: replace protobuf submodule with vcpkg#162
wenchy merged 67 commits into
masterfrom
vcpkg

Conversation

@Kybxd

@Kybxd Kybxd commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace the third_party/_submodules/protobuf build pipeline with a vcpkg-based flow, and replace the Windows-only prepare.bat with a cross-platform make.py orchestrator. Same setup/build/test commands on Windows, macOS, Linux, and inside the devcontainer; same flow on CI.

Key changes

Toolchain

  • Drop third_party/_submodules/protobuf + init.{sh,bat}. Protobuf comes from vcpkg pinned to VCPKG_BASELINE_COMMIT in .devcontainer/versions.env (single source of truth: Dockerfile, make.py, CI all read from it).
  • test/cpp-tableau-loader/CMakeLists.txt uses find_package(Protobuf CONFIG REQUIRED) only.

make.py (new)

  • Single Python 3.10+ stdlib-only entrypoint at the repo root.
  • Subcommands: setup / generate / build / test / clean / env.
  • Pins every toolchain dimension on every native host: Go via go.dev tarball, buf via GitHub release, protobuf via vcpkg on Windows + macOS + Linux + devcontainer, .NET via Homebrew/Microsoft repo, cmake/ninja via host package manager.
  • Windows: each subprocess sources vcvarsall.bat itself; shell PATH/INCLUDE/LIB never mutated.
  • --protobuf-version switches to vcpkg manifest mode for legacy/version-specific builds — no re-running setup required.
  • Dry-run / verbose / cwd flags. Resolved paths cached in ~/.loader-env.json.

Tests for make.py itself

  • test_make.py (next to make.py) — pytest unit tests + dry-run snapshot tests.
  • New testing-make.yml workflow runs them on ubuntu/macos/windows on every push (<2 min).

CI

  • testing-cpp.yml: matrix over modern (protobuf 6.33.4) × legacy-v3 (3.21.12), ubuntu × windows. lukka/run-vcpkg for cached vcpkg. Build/test step is python make.py test --lang cpp ….
  • testing-go.yml: python make.py test --lang go --race --coverage.
  • testing-csharp.yml: single modern protoc (libprotobuf is irrelevant for C#); python make.py test --lang csharp.
  • devcontainer-smoke.yml: bind-mount + python3 make.py test --lang go --smoke (amd64 + arm64).

Docs

  • README, CLAUDE.md, .devcontainer/README.md collapsed to make.py-first usage. prepare.bat removed.

Test plan

  • All 5 CI workflows green on Linux + macOS + Windows.
  • python make.py test --lang cpp (modern 6.33.4) — 13/13 ctest cases pass on Windows native.
  • python make.py test --lang cpp --protobuf-version 3.21.12 — 13/13 pass (manifest mode).
  • python make.py test --lang go and --lang csharp — full suites pass.
  • pytest test_make.py — 82+ unit + dry-run tests, <5s.

Migration

git submodule deinit -f third_party/_submodules/protobuf
rm -rf third_party/_submodules/protobuf .git/modules/third_party/_submodules/protobuf
python make.py setup --lang all

🤖 Generated with Claude Code

Kybxd added 9 commits May 28, 2026 15:50
Switch from building the protobuf submodule via init.sh/init.bat to
installing protobuf through vcpkg. This removes the bundled protobuf
build, simplifying the setup process and significantly reducing initial
bootstrap time.

Update prepare.bat to install vcpkg and protobuf (x64-windows-static),
expose the vcpkg-built protoc on PATH, and export VCPKG_ROOT. Revise
README.md instructions to document vcpkg (recommended), system package,
Homebrew, and source-based protobuf installation paths. Adjust gitignore
rules for the vcpkg manifest and install tree accordingly.
@Kybxd Kybxd changed the title ci: migrate protobuf install to vcpkg in CI refactor: replace protobuf submodule with vcpkg May 28, 2026
wenchy and others added 20 commits May 29, 2026 21:27
- prepare.bat: switch to vcpkg manifest mode when PROTOBUF_VCPKG_VERSION
  is set. Classic-mode `--x-version` is silently a no-op, so the previous
  pin attempt would lie to users. Render a vcpkg.json under
  %LOCALAPPDATA%\loader\vcpkg-manifest\ with builtin-baseline + overrides,
  install via `vcpkg install --x-install-root=...`, and post-assert that
  the resolved port version matches the request. Pin the vcpkg checkout
  itself to VCPKG_BASELINE_COMMIT (mirroring testing-cpp.yml's VCPKG_COMMIT)
  so classic mode is also reproducible. Drop `--depth 1` so the commit pin
  is reachable. Export VCPKG_INSTALLED_DIR for downstream cmake.
- prepare.bat: add a `where cl.exe` preflight at the top of Step 5 so an
  unactivated MSVC environment fails fast with an actionable message
  instead of cryptic vcpkg compiler-detection errors.
- testing-cpp.yml: replace deprecated `version-string` with `version` in
  the rendered vcpkg.json.
- testing-csharp.yml: document inline that `protobuf-version` is the
  protoc release tag (e.g. 33.4) and explain how it maps to the C++
  libprotobuf SemVer used in testing-cpp.yml.
- README.md: add a "Migrating from the bundled-protobuf layout" callout
  with the submodule-deinit recipe so existing checkouts know how to
  clean up. Replace the misleading classic-mode `--x-version` snippet
  with a working manifest-mode example. Document the extra cmake flags
  (-DVCPKG_INSTALLED_DIR / -DVCPKG_MANIFEST_INSTALL=OFF) required when
  PROTOBUF_VCPKG_VERSION is used.
- .gitignore: ignore .claude/settings.local.json.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the brainstormed design for adding a Dev Container under
.devcontainer/ to give contributors a one-command, reproducible
multi-language toolchain (C++17 + Go 1.24 + .NET 8 + Node 20 + buf
1.67.0 + protobuf 6.33.4 via vcpkg, all pinned to CI's exact versions).

Key decisions captured:
- Single all-in-one Ubuntu 24.04 image (covers all four languages)
- Dockerfile in repo, build on-demand (no ghcr.io publish in v1)
- Multi-arch native via TARGETARCH (amd64 + arm64; no QEMU on Apple Silicon)
- Pinnable protobuf version via LOADER_PROTOBUF_VERSION host env var,
  flowing through devcontainer.json build args into vcpkg manifest mode
- Devcontainer is recommended path; prepare.bat / per-language manual
  setup stays as fallback for contributors who can't run Docker
- CI keeps lukka/run-vcpkg directly (devcontainer is not used in CI)

Implementation plan to follow via writing-plans.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12-task plan implementing the design at
docs/superpowers/specs/2026-05-29-devcontainer-design.md.
Each task is a single Dockerfile layer or config file with
build/verify/commit steps and concrete expected outputs.

Refs: docs/superpowers/specs/2026-05-29-devcontainer-design.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bootstrap the devcontainer image with Microsoft's multi-arch
mcr.microsoft.com/devcontainers/cpp:1-ubuntu-24.04 base. Subsequent
commits layer Go, buf, vcpkg/protobuf, .NET, and Node on top.

Refs: docs/superpowers/specs/2026-05-29-devcontainer-design.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves TARGETARCH (amd64 or arm64) into per-arch values
(Go tarball arch, buf release-asset arch, vcpkg triplet) and
writes them to /opt/buildargs.env for downstream RUN layers
to source. Unknown arches fail the build.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install Go from the official multi-arch tarball into /usr/local/go.
PATH is exposed via ENV (not /etc/profile.d) so non-interactive shells
(postCreateCommand, downstream RUNs) see `go` without sourcing profile.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-binary release into /usr/local/bin/buf. Pinned to the same
version testing-cpp.yml / testing-csharp.yml use.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pin vcpkg to commit dc8d75c…df932 (lock-step with prepare.bat and
testing-cpp.yml's VCPKG_COMMIT). Render a minimal vcpkg.json manifest
with the protobuf override + builtin-baseline, install via
manifest mode (the only mode where the version pin actually takes
effect), and post-assert that the resolved port version starts with
the requested PROTOBUF_VERSION. Default is 6.33.4; legacy v3 reachable
via --build-arg PROTOBUF_VERSION=3.21.12.

Symlink /opt/vcpkg/active → installed/<triplet> and /usr/local/bin/protoc
→ active/tools/protobuf/protoc so downstream ENV/PATH stays
arch-independent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Microsoft apt repo for dotnet-sdk-8.0 (matches testing-csharp.yml's
target). NodeSource setup_20.x for Node 20 LTS (covers the
experimental _lab/ts/ workflow; not currently in CI). Both clean
their apt caches to keep the layer small.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CMAKE_PREFIX_PATH=/opt/vcpkg/active lets the existing README cmake
recipe (-DCMAKE_BUILD_TYPE=Debug, no toolchain file) resolve protobuf
inside the container without any flag changes from the host workflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the Dockerfile under build.args, mounts a named volume for the
Go module cache, declares the VS Code extension set, and prints a
one-line ready-banner via postCreateCommand. PROTOBUF_VERSION flows
from the host LOADER_PROTOBUF_VERSION env var (default 6.33.4) so
contributors can rebuild against the legacy v3 line via:
  LOADER_PROTOBUF_VERSION=3.21.12 code .

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-pager covering prerequisites (Docker Desktop / Engine + VS Code
Dev Containers extension), how to open the container, the
LOADER_PROTOBUF_VERSION knob, host-OS caveats (WSL2 workspace
location, Apple Silicon native arm64), and the layered Dockerfile
architecture. Points users at prepare.bat / per-language manual setup
as the explicit fallback path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a new "Recommended: Dev Container (any host OS)" subsection at the
top of Prerequisites pointing contributors at .devcontainer/. Add a
"Skip this section if you're using the devcontainer" lead-in to the
existing "Install protobuf" and "Windows: bootstrap" blocks so the
manual paths are clearly the fallback, not the primary route.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributors with an existing host checkout can hit confusing build
errors when stale .pb.cc / .pb.cs files from a previous protoc-3.x
session linger in the workspace (gitignored, so git pull doesn't
remove them). The container's modern protoc 33.4 emits a different
file layout, and the leftovers shadow what's freshly generated. Add
a Troubleshooting section with the rm -rf recipe to wipe and retry.
A fresh clone doesn't hit this.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…TODO

The Troubleshooting recipe added in 8a583ba targeted directories that
don't actually hold stale generated files: src/tableau is empty (real
C++ stale files live at src/protoconf/tableau/), and protoconf/tableau
doesn't exist (C# protoc emits flat .cs files into protoconf/). Fix
the rm -rf paths so the recipe actually unblocks the failure mode it
documents.

Also drop the now-contradictory "> TODO: [devcontainer]" placeholder
from README.md line 7 — the devcontainer is implemented and recommended
in the section immediately below it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nfile cache

Docker creates named volumes as root:root by default, which leaves the
Go module cache mounted at /home/vscode/go unwritable for the `vscode`
user inside the devcontainer. Chown it to `vscode:vscode` in
postCreateCommand so `go mod download` / `go build` work out of the box.

Also ignore dotnet/runfile-discovery/, an auto-generated .NET SDK 10
file-based app discovery cache produced by the dotnet CLI / C# Dev Kit.
Update test/buf.yaml dependency pin and refresh test/buf.lock with the
new commit / digest.
wenchy and others added 18 commits June 4, 2026 19:01
Two fixes for testing-{make,cpp}.yml regressions:

1. Platform.cmake_toolchain_args() gains force_vcpkg=True. When set,
   emits -DCMAKE_TOOLCHAIN_FILE=...vcpkg.cmake + -DVCPKG_TARGET_TRIPLET
   on every host, not just Windows. Used by _cpp_build_or_test when
   --protobuf-version is set: manifest mode means we ARE using vcpkg
   regardless of OS, so cmake's find_package(Protobuf) needs the
   toolchain to resolve against vcpkg_installed/.
   Fixes testing-cpp.yml ubuntu-latest legacy-v3 failure ("Could not
   find a package configuration file provided by Protobuf").

2. _cpp_build_or_test no longer aborts in --dry-run when VCPKG_ROOT
   is unset. It substitutes a placeholder so snapshot tests still
   verify the printed command sequence on hosts (CI runners) without
   vcpkg installed. Real (non-dry-run) execution still hard-errors.
   Fixes testing-make.yml ubuntu/macos failures
   (test_test_lang_cpp_protobuf_version_manifest_mode + sibling).

Adds two regression tests:
- test_test_lang_cpp_manifest_no_vcpkg_root_dry_run_ok  (delenv VCPKG_ROOT)
- test_test_lang_cpp_manifest_forces_toolchain_on_linux (asserts -DCMAKE_TOOLCHAIN_FILE in output)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous value `dc8d75cfc3281b8e2a4ed8ee4163c891190df932` was the
SHA of an *annotated tag object* (release/2026.04.27), not a commit.
Symptoms:
  - github.com/microsoft/vcpkg/commit/<tag-SHA> 404s.
  - `git log` doesn't list it (tag objects aren't commits).
  - `git cat-file -t` reports "tag", not "commit".
Functionally everything worked (git/vcpkg auto-peel tag SHAs to commits),
but it's unbrowsable on GitHub and confuses anyone trying to verify the
pin. Switched to the underlying commit SHA so the GitHub /commit/<sha>
view resolves; the resolved port catalog is identical.

Also fixes a pre-existing bug uncovered while validating the SHA change:
classic-mode `make.py test --lang cpp` (no --protobuf-version) didn't
remove a stale vcpkg.json left over from a previous manifest-mode run.
cmake's vcpkg toolchain auto-detected the manifest and built the wrong
libprotobuf into `build/vcpkg_installed/`, mismatching the
`buf generate`-produced .pb.h files (`Cannot open include file:
google/protobuf/runtime_version.h`). Now the cpp handler unlinks any
leftover vcpkg.json before configure when not in manifest mode.

Workflow YAMLs in this commit are pure formatter normalization
(quote style + indent), no semantic change.

Verified: 74/74 unit tests pass; full C++ build + 13/13 ctest cases
green on Windows against the new SHA.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…where)

Until now `make.py setup --lang cpp` on macOS/Linux installed whatever
`brew` / `apt` / `dnf` shipped — meaning a fresh dev machine in 2027
might pick up protobuf 7.x while CI is still pinned to 6.33.4. Now
mirrors the devcontainer Dockerfile's strategy across every native host:

  - Go: official tarball from go.dev to ~/.local/go/ (Linux/macOS).
  - buf: GitHub release binary at BUF_VERSION.
  - protobuf: vcpkg at VCPKG_BASELINE_COMMIT, classic mode by default.
    Manifest mode (--protobuf-version) still works as before.
  - cmake / ninja / build-essential: distro/brew package (version-tolerant).
  - .NET: Homebrew dotnet@N (macOS), Microsoft apt repo (Linux).
  - Node: Homebrew node@N (macOS), NodeSource (Linux).

Internal refactor:
  - _setup_vcpkg_windows lifted to cross-platform _setup_vcpkg
    (.bat vs .sh bootstrap, MSVC wrap on Windows only, exe vs binary).
  - Platform.cmake_toolchain_args() emits toolchain flags whenever
    vcpkg_root is known, regardless of OS (was Windows-only). Devcontainer
    still returns [] (CMAKE_PREFIX_PATH preset).
  - hydrate_platform_from_env reads ~/.loader-env.json on every OS now
    (was Windows-only), so `make.py test` after `make.py setup` works
    without VCPKG_ROOT being on shell PATH.
  - _ensure_buf_linux generalized to _ensure_buf_unix (Linux/Darwin).
  - New _ensure_go_tarball matches devcontainer's Go install.
  - force_vcpkg parameter on cmake_toolchain_args is no longer needed
    in practice (default behaviour already does the right thing) but
    kept as an explicit override for callers.

8 new regression tests (TestCrossPlatformPinning + expanded
TestPlatform.cmake_toolchain_args*). 82/82 pass. Real Windows C++ build
+ 13/13 ctest cases still green against the refactored code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pinning protobuf via 'overrides' alone leaves builtin-baseline on a newer
vcpkg commit, which lets transitive deps (abseil/utf8-range/re2/...) drift
forward and breaks ABI (e.g. missing absl::if_constexpr). Pin the baseline
itself to the vcpkg commit whose versions/baseline.json had the target
protobuf — that snapshot is by construction self-consistent.

make.py:
  - Add _resolve_vcpkg_baseline_for_protobuf: pickaxe-search vcpkg's git
    history (git log -S '"X.Y.Z"' -- versions/baseline.json), validate
    each candidate via baseline.json (default.protobuf.baseline == X.Y.Z)
    to defeat false positives, cache result in ~/.loader-env.json.
  - Add --vcpkg-baseline override; auto-resolve from $VCPKG_ROOT when
    only --protobuf-version is given. Dry-run returns a placeholder so
    snapshot tests stay deterministic.
  - Drop 'overrides' from rendered vcpkg.json — baseline alone now carries
    protobuf + transitive deps as one consistent set.

CI (.github/workflows/testing-cpp.yml):
  - Move vcpkg-commit into matrix.config alongside protobuf-version so
    each row pins its own self-consistent snapshot.
  - Pass --vcpkg-baseline explicitly to make.py: lukka/run-vcpkg checks
    out vcpkg shallowly, so the runtime resolver can't pickaxe locally.
  - Render vcpkg.json baseline-only (no 'overrides') with the same SHA
    we hand to lukka/run-vcpkg, keeping cache key and final build aligned.
lukka/run-vcpkg@v11 auto-injects VCPKG_BINARY_SOURCES=clear;x-gha,readwrite,
but the 'x-gha' provider only exists in vcpkg ≳ 2023. The legacy-v3 matrix
row pins a 2022-era vcpkg snapshot (6245ce44..., for protobuf 3.21.12) whose
vcpkg binary errors out with:

    unknown binary provider type: ... on expression: clear;x-gha,readwrite

lukka/run-vcpkg honours a pre-set VCPKG_BINARY_SOURCES, so set it to 'clear'
on the step. We don't actually lose caching: actions/cache@v4 above already
keys vcpkg_installed/ on (os, triplet, vcpkg-commit, vcpkg.json hash) and
restores it before lukka/run-vcpkg runs.
The 2023-01-15 vcpkg snapshot pinned by legacy-v3 still references MSYS2
packages whose URLs and SHA512s have since been pruned from repo.msys2.org
(MSYS2 doesn't keep older versions). protobuf:x64-windows pulls them in via
vcpkg_acquire_msys → vcpkg_fixup_pkgconfig, so the install fails:

    Failed to download file with error: 1
    ... vcpkg-scripts version: 6245ce44a0 2023-01-15 (3 years, 5 months ago)

Linux is unaffected because it uses the system pkg-config. The modern matrix
row still exercises Windows on a current vcpkg snapshot, so we keep legacy-v3
as a Linux-only smoke test for the protobuf-3 ABI.
Captures the design for two cleanups surfaced while testing
make.py setup --lang all on a fresh ubuntu:24.04: drop the TS
language axis (CLAUDE.md already calls it experimental, no CI
references it) and tighten the devcontainer detection to
/opt/vcpkg/active only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/.dockerenv is created by Docker for every container, so the previous
`exists() or exists()` heuristic silently no-op'd `make.py setup` in
any plain Docker image (e.g. ubuntu:24.04). Use only the marker the
devcontainer's Dockerfile actually sets.

Adds two TestPlatform cases that monkeypatch Path.exists to lock in
both the negative (/.dockerenv alone -> False) and positive
(/opt/vcpkg/active -> True) signals.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CLAUDE.md already documents TS as experimental and not in CI; no
workflow references it. Rather than maintain a broken pipeline
(_ensure_node_linux was a no-op on Linux, _lab/ts/package.json has
no build script), shrink the language axis to (go, cpp, csharp).

Removes:
  - LANGS_ALL "ts" entry (argparse choices update transitively)
  - Versions.node_version property
  - _ensure_node_linux (silent no-op)
  - per-platform Node/npm install branches in _setup_{linux,macos,windows}
  - _ts_build_or_test and its _build_or_test dispatch
  - _lang_dir, generate, cmd_clean, cmd_env "ts"/node/npm arms
  - module docstring `npm test` mention and --lang ts examples

The _lab/ts/ scratchpad stays in the tree for manual experimentation
but is no longer driven by make.py.

test_make.py loses NODE_VERSION/node_version assertions and the
test_ts_lives_under_lab case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to dropping TS support in make.py — NODE_VERSION is no
longer consumed by any tooling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
make.py no longer drives any TS pipeline (see prior commit), so the
devcontainer image no longer needs Node. Also trims comment headers
that named Node.

Triggers a one-time devcontainer rebuild for users with the cached
image.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to dropping Node from the Dockerfile — banner script
called `node --version` which would fail at container startup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
make.py no longer supports --lang ts; the TS scratchpad at _lab/ts
remains in-tree but is unmaintained and not surfaced in the docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Windows `Path("/opt/vcpkg/active")` renders as
`\opt\vcpkg\active` via str(), so the mock never matched the
forward-slash literal and `test_detect_recognizes_vcpkg_active_marker`
failed on windows-latest. `Path.as_posix()` returns the
forward-slash form on every platform.

Also fixes `test_detect_ignores_dockerenv_marker` which was passing
on Windows for the wrong reason (mock never returned True; assertion
expected False, so it accidentally passed).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression from dropping the NodeSource curl: that script ran its own
`apt-get update` as a side effect, which refreshed the cache after the
preceding `dpkg -i packages-microsoft-prod.deb` added Microsoft's
apt source. Removing it left no update between adding the repo and
installing dotnet-sdk-${DOTNET_VERSION}, so apt couldn't find the
package and the build failed with exit 100.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
versions.env now declares one (protobuf, vcpkg-baseline) row per variant
(MODERN_*, LEGACY_V3_*) plus a DEFAULT_VARIANT pointer, replacing the
flat single-row PROTOBUF_VERSION + VCPKG_BASELINE_COMMIT keys. This:

  - eliminates the drift between versions.env's modern vcpkg SHA
    (was 56bb2411) and testing-cpp.yml's modern SHA (was 1f6bbba3) —
    they now both pin 56bb2411 (tip of vcpkg's 2026.04.27 quarterly)
  - brings the legacy-v3 pair (protobuf 3.21.12 / vcpkg 6245ce44)
    into versions.env so it's discoverable outside the CI YAML
  - adds a clean knob to switch the entire row at once via
    LOADER_DEFAULT_VARIANT host env -> DEFAULT_VARIANT build-arg ->
    Dockerfile case statement

Consumers all learn the resolver:

  * make.py Versions class: new default_variant + variants() accessors;
    existing protobuf_version / vcpkg_baseline_commit properties
    transparently resolve through DEFAULT_VARIANT (with fallback to
    unprefixed keys for backward compat)
  * .devcontainer/Dockerfile: case statement after sourcing versions.env
    re-exports PROTOBUF_VERSION / VCPKG_BASELINE_COMMIT for the rest
    of the build script
  * .devcontainer/devcontainer.json: new LOADER_DEFAULT_VARIANT host
    env mapping (alongside the existing LOADER_PROTOBUF_VERSION
    surgical override)
  * .github/actions/load-versions: same case logic, exports the
    resolved unprefixed names to GITHUB_ENV so existing workflow
    references keep working
  * .github/workflows/testing-cpp.yml: modern row's vcpkg-commit synced
    to versions.env, comment now points readers at versions.env

test_make.py: +4 new TestVersions cases covering default_variant
fallback, variants() enumeration, dash-vs-underscore label parsing,
and unprefixed-key backward compat. All 87 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kybxd added 2 commits June 8, 2026 16:30
… compat

- embed/util.pc.h: introduce util::GetExtension() wrapper that works around
  the protobuf v3.15.0 GetExtension reflection bug by falling back to
  reparse-via-FieldDescriptor when GOOGLE_PROTOBUF_VERSION < 3015000.
- embed/util.pc.cc, embed/load.pc.cc: switch the two extension lookups
  (tableau::field on field options, tableau::worksheet on message options)
  to use util::GetExtension() instead of calling .GetExtension() directly.
- test/cpp-tableau-loader/src/protoconf/*: regenerate the mirrored sources
  to stay in sync with the embed templates.
- make.py: vcpkg-based protobuf integration for cpp-loader testing.
Kybxd and others added 6 commits June 8, 2026 18:04
- Add MapKeyFd helper: falls back to field(0) on protobuf < v3.12.0 where Descriptor::map_key() is unavailable (map-entry key is always field 0 per proto3 wire-format contract).

- Rework GetExtension to work around a static-initialization-order quirk on runtimes < v3.15.0 by serializing and reparsing into a fresh OptionsT; unify the signature with a single return type across branches.

- Drop redundant util:: qualifier on GetExtension call inside tableau::util::PatchMessage.
…rces

- go: drop the hand-maintained golangKeywords map and detect Go keywords via go/token (token.Lookup(...).IsKeyword()), matching protoc-gen-go's GoSanitized; keep the loader-specific "x" receiver-name guard. Also removes a stale bogus "pi" entry.

- cpp: align cppKeywords with protoc's kKeywordList and point the reference at protobuf's compiler/cpp/helpers.cc.

- csharp: clarify that protobuf's C# generator keeps no language-keyword list (PascalCase + reserved member names), so the loader maintains the C# keyword set itself; reference csharp_helpers.cc GetPropertyName and the C# spec.
Update every `python make.py ...` example across docs, CI workflows,
and make.py's own usage docstring + error messages to use `python3`,
matching the file's `#!/usr/bin/env python3` shebang and the Python
>= 3.10 requirement called out in the module docstring.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ns.env

Windows setup previously gated the Go install on `"go" in langs` and
hardcoded the winget ids `GoLang.Go.1.24` / `Microsoft.DotNet.SDK.8`.
Two issues:

1. Every `buf generate` invokes the Go protoc plugins via
   `go run ../../cmd/protoc-gen-...` (per `buf.gen.yaml`), so Go is
   required regardless of which language is being built. With the gate
   in place, `setup --lang cpp` and `setup --lang csharp` left a Windows
   host without Go and every codegen step failed.

2. The hardcoded winget ids drifted from versions.env on bumps,
   breaking the "single source of truth" contract macOS/Linux already
   satisfy (their tarball/winget paths read ctx.versions.go_version /
   .dotnet_version).

Drop the gate and derive both ids from versions.env:
  GoLang.Go.<MAJOR>.<MINOR>          from GO_VERSION
  Microsoft.DotNet.SDK.<MAJOR>       from DOTNET_VERSION

Add `TestSetupWindows` covering both invariants. Tests use synthetic
GO_VERSION=1.99.7 / DOTNET_VERSION=9.0 so they catch a regression even
though the real versions.env (1.24.0 / 8.0) happens to match the old
hardcoded values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pport floor

Extend vcpkg protobuf pinning to handle versions older than vcpkg's
baseline.json floor (3.14.0): pin builtin-baseline at the floor commit
and add an `overrides` entry for the older protobuf. The ABI-drift
hazard does not apply since pre-3.14 protobuf is dependency-light.

Add a distinct hard *support* floor (3.8.0): below it the C++ loader is
known not to compile, so refuse such versions up-front (before touching
user trees) with a clear RuntimeError, unless -f/--force is passed to
investigate the breakage. The guard is a pure version-number check and
runs before baseline resolution and the stale-codegen wipe.

Improve the "version not present" hint: when listing below-baseline
versions, enumerate the full range down to the 3.8.0 support floor
without truncation (was capped, hiding older entries as "+N older"),
and drop versions below the floor since they can't compile anyway.

Add TestProtobufMinSupported and TestKnownVersionsHintFloor covering the
floor guard, --force behavior, and the down-to-3.8.0 hint.
@wenchy
wenchy merged commit e96beba into master Jun 9, 2026
12 checks passed
@wenchy
wenchy deleted the vcpkg branch June 9, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants