Open
Conversation
Move the build configuration to the repository root so that 'pip install .' compiles all library C sources directly into the Python extension module, removing the need for a pre-built libbpak shared library. - Add root setup.py with all lib and wrapper sources - Add pyproject.toml with setuptools build-system declaration - Add MANIFEST.in for sdist packaging - Add python/bpak_user_settings.h using the existing BPAK_HAVE_USER_SETTINGS mechanism in bpak.h - Remove old python/setup.py that required libbpak
- Add pip install instructions to README.rst and docs/build.rst - Fix stale cmake commands in docs to use out-of-tree builds - Remove non-existent BPAK_BUILD_PYTHON_WRAPPER from docs and cmake options table, add BPAK_BUILD_TOOL - Fix test/CMakeLists.txt to guard Python tests with BPAK_BUILD_TESTS instead of undefined BPAK_BUILD_PYTHON_WRAPPER - Update examples/python/Dockerfile from stale autotools to pip install - Add *.egg-info/ to .gitignore
Add wrapper bindings the Python bpak CLI relies on: - python_wrapper.c: hash_kind_str, signature_kind_str, id_to_string, meta_to_string, add_transport_meta, parse_public_key. - package.c: flags parameter on add_file/add_key, extract_file, delete_all_parts, part_sha256, and bounds checking on add_meta to refuse oversized payloads up front. - part.c: flags / transport_size / pad_bytes getters, and keep_meta keyword on Part.delete.
Add the bpak Python package and a Click-based CLI whose commands, flags, positional arguments, and mutually-exclusive groups match the C bpak binary in src/ exactly. Existing scripts and muscle memory keep working. Notable parity choices: - transport: single command with -a/-E/-D mode flags. --part is exposed as a long-only alias of --part-ref to match the long-option abbreviation behaviour of getopt_long in src/transport.c. - generate: positional dispatch (id, keystore). Unknown generators exit with an error instead of silently no-op'ing as in C. - show: -p reinterprets as the metadata part_id_ref filter when -m is given, matching src/show.c. - -v/--verbose lives on each subcommand (not a global flag), mirroring per-action getopt parsing in C. The new _apply_verbose helper resets the C extension's static log callback on level 0 so verbose state does not leak across calls in the same process. - Version banner is "BitPacker <ver>" to match src/misc.c. setup.py: declare the bpak package, rename the extension to bpak._bpak, register the bpak console_scripts entry point, and require Click >= 8.0.
The Python package was restructured for pip install (commit 577abc2), but the CMake build kept producing a legacy monolithic `bpak.so`. That left `sys.path.insert("../python/"); import bpak` in the test_python_*.py harness unable to resolve the compiled module (`PyInit_bpak` was never exposed; the init function is `PyInit__bpak`), so ctest reported `ImportError: dynamic module does not define module export function` for every Python test on this branch. Stage the pure-Python package sources into ${CMAKE_CURRENT_BINARY_DIR}/bpak/ and build the CPython extension as `_bpak.so` alongside them, matching the `bpak._bpak` layout that setup.py already produces. Switches to Python_add_library(MODULE ...) for a proper Python extension target instead of a generic shared library. With this in place the existing Python tests run under ctest again, and the new test_python_cli.py picks up the same layout.
Replace the 825-line argv-for-argv port of the C bpak binary with a
split Click application under bpak._cli/. The Python CLI is no longer a
mirror of the C argv; it is the best idiomatic shape for a Click tool,
even at the cost of breaking compatibility with prior Python-bpak scripts.
The C bpak binary is untouched.
Shape (two-level command groups):
bpak create FILE [--hash ...] [--signature ...] [--force]
bpak compare FILE1 FILE2
bpak show FILE (full overview)
bpak show meta FILE [ID] [--part-ref REF]
bpak show part FILE ID [--hash]
bpak show hash FILE [--binary]
bpak add part FILE ID --from PATH [--no-hash]
bpak add meta FILE ID (--from-string V | --from-file P)
[--encoder ...] [--part-ref REF]
bpak add key FILE ID --from PATH
bpak add merkle FILE ID --from PATH
bpak set meta FILE ID VALUE [--encoder ...] [--part-ref REF]
bpak set header FILE [--key-id ID] [--keystore-id ID]
bpak delete part FILE (ID | --all) [--keep-meta]
bpak delete meta FILE ID [--part-ref REF]
bpak extract part FILE ID [--output PATH]
bpak extract meta FILE ID [--output PATH] [--part-ref REF]
bpak sign FILE (--key PATH | --signature PATH)
bpak verify FILE (--key PATH | --keystore PATH)
bpak transport add FILE ID --encoder N --decoder N
bpak transport encode FILE --output PATH [--origin PATH]
bpak transport decode FILE --output PATH [--origin PATH]
bpak generate id STRING
bpak generate keystore FILE --name NAME [--decorate]
Correctness fixes rolled in with the rewrite:
- compare now diffs part contents via Package.part_sha256 on both sides
(plus part-header fields), replacing the previous size-only check that
reported same-size-different-content parts as equal.
- show's full overview prints a single "Header hash" line; the old code
mis-labelled the same Package.digest value as both header and payload.
- sign / verify enforce exactly-one-of their key options; the previous
CLI silently accepted neither being given.
- verify opens the package read-only, so it works on read-only packages.
- generate keystore rejects --name values that aren't valid C identifiers
and validates the keystore-provider-id metadata length before unpacking.
- Commands that write binary to stdout (extract part/meta, show hash
--binary) refuse to run when stdout is a TTY, preventing terminal
corruption.
Shared plumbing in bpak/_cli/_common.py:
- BpakId ParamType (replaces scattered resolve_id calls).
- @open_package decorator centralises the Package open/close + error
translation every command repeated by hand.
- exactly_one_of / at_least_one_of / incompatible helpers use an is_set
predicate so "--key-id 0" counts as provided, not missing.
- install_verbose wires the _bpak log callback at the root group entry
and tears it down via ctx.call_on_close; log output always goes to
stderr so stdout stays clean for scripting.
- binary_sink(path) centralises the TTY-refusal guard.
- safe_c_identifier validates generate-keystore --name against
[A-Za-z_][A-Za-z0-9_]*.
_helpers.resolve_id now also parses bare decimal literals, so
`--part-ref 0` means the integer 0 rather than CRC32("0"); this matches
the C wrapper's part_id_ref default.
__main__.py becomes a three-line shim. setup.py ships bpak._cli and
repoints the console_scripts entry at bpak._cli:cli.
test/test_python_cli.py drives every subcommand of the new bpak._cli through click.testing.CliRunner, including the happy path per subcommand, mutex errors (exactly_one_of / at_least_one_of), binary-to-TTY refusal, same-size-different-content part detection in compare, zero-value validation (--key-id 0), show meta --part-ref filtering, a sign/verify round trip on a read-only package, a transport add/encode/decode round trip, and generate keystore --name identifier rejection. Registered in test/CMakeLists.txt so it runs under ctest alongside the existing Python tests. docs/ug/99_python_cli.rst documents the new surface with a full command reference, notable differences from the C CLI, a migration table mapping the previous Python argv to the new shape, and a list of breaking changes (no compatibility shims are provided). A note at the top of 01_basics.rst points Python-installed users to the new page since the existing user-guide examples document the C bpak binary.
Project metadata, dependency-groups, and initial ruff/mypy config for the CLI package. Ruff runs with select = ["ALL"]; the ignore list silences high-noise/low-value rules for this codebase (TRY003/EM10x raise-message churn, D203/D213 pydocstyle conflicts, ERA001 commented-out-code false positives). Per-file ignores let the .pyi stub mirror the C API's `id`/`input` parameter names, the _cli __init__ keep its circular-import workaround, and the test/ scripts use assert/print/no-docstrings freely.
Hand-written .pyi declaring Package (context-manager; sign/verify/ add_file/add_meta/get_part/get_meta/extract_file/delete_all_parts/ part_sha256/verify_with_keystore + getset attrs), Part, Meta, Error, module-level constants, and the module-level functions (id, id_to_string, hash_kind_str, signature_kind_str, meta_to_string, add_transport_meta, parse_public_key, transport_encode/decode, set_log_func). Signatures sourced from python/python_wrapper.c, package.c, part.c, and meta.c. Add __all__ to python/bpak/__init__.py so mypy and ruff see the re-exported surface explicitly rather than treating every import as unused.
Type the decorators in _cli/_common.py with ParamSpec/TypeVar/ Concatenate so open_package and handle_bpak_errors preserve their wrapped callback's signatures (filename -> pkg substitution for open_package). Annotate every @open_package-decorated callback as taking pkg: _bpak.Package, now possible with the new .pyi stub. Narrow str | None option values with assert after exactly_one_of resolves the choice in sign.py, add.py, compare.py, and delete.py; mypy can't see through the dict-based validator, and the asserts document the invariant. Absolute imports throughout: replace `from ..` / `from .._helpers` with `from bpak import _bpak` / `from bpak._helpers import …`. Move type-checking-only imports (collections.abc.Callable, _bpak) under TYPE_CHECKING where appropriate. Miscellaneous: drop unnecessary elif-after-return in _helpers.py; use Path.exists()/Path.open() in create.py/extract.py/_common.py; replace blind Exception catches with PackageNotFoundError in _cli/ __init__.py and generate.py; tidy docstrings; run ruff format across the package. After this change ruff check, ruff format --check, and mypy all pass on python/bpak with zero issues (from 134 ruff + 6 mypy errors).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This does a similar rewrite to what was done in Punchboot to get bpak directly installable by "pip install".
Also modernized cli to click. The cli rewrite is done in two steps, first introduce a Python cli that mirrors the C cli, and then rewrite it to be more idiomatic.