Skip to content

feat(python): add PyO3-based Python bindings#600

Merged
dmtrKovalenko merged 15 commits into
dmtrKovalenko:mainfrom
4fuu:feat/python-bindings
Jun 17, 2026
Merged

feat(python): add PyO3-based Python bindings#600
dmtrKovalenko merged 15 commits into
dmtrKovalenko:mainfrom
4fuu:feat/python-bindings

Conversation

@4fuu

@4fuu 4fuu commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds a new Python binding for FFF using PyO3, providing a native extension with a clean Pythonic API.

Changes

  • New Rust crate crates/fff-python/ with PyO3 bindings
    • FileFinder class exposing search, glob, directory search, mixed search, grep and multi-grep APIs
    • Full result types: FileItem, DirItem, GrepMatch, Score, etc.
    • Lifecycle support via context manager and destroy()
  • Python package packages/fff-python/ using the maturin build backend
    • PyPI package name is fff-python; import name remains fff
    • Updated README, tests and standalone example
  • CI
    • Added .github/workflows/python.yml for Ubuntu/macOS/Windows testing
    • Extended release.yaml to build Python wheels (x86_64/aarch64) and sdist, with optional PyPI publishing via trusted publishing
  • Workspace
    • Added crates/fff-python to workspace members and updated Cargo.lock
    • Updated .gitignore for Python build artifacts

Verification

Tested on the fork repository; the relevant CI checks pass:

  • cargo check -p fff-python
  • cargo clippy -p fff-python --no-deps -- -D warnings
  • uv run pytest -v ✅ (8 passed)
  • uv run python examples/basic.py .
  • Python CI passes on Ubuntu, macOS and Windows
  • Prebuild Python wheel jobs build successfully across all target platforms

4fuu added 2 commits June 14, 2026 19:02
- Add native PyO3 extension in crates/fff-python
- Replace ctypes wrapper with maturin-built package in packages/fff-python
- Expose FileFinder, search/glob/directory/mixed/grep APIs and result types
- Use PyPI package name fff-python (import name remains fff)
- Update workspace Cargo.toml/Cargo.lock and .gitignore for Python artifacts
- Add .github/workflows/python.yml to test bindings on Ubuntu/macOS/Windows
- Extend release.yaml with Python wheel builds (x86_64/aarch64) and sdist
- Add optional PyPI publish job using trusted publishing
@4fuu 4fuu changed the title feat(python): migrate Python bindings from ctypes to PyO3 feat(python): add PyO3-based Python bindings Jun 14, 2026

@dmtrKovalenko dmtrKovalenko left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks good to me (Im not a good python dev lol)

@gustav-fff can you also review for any issues

@dmtrKovalenko

Copy link
Copy Markdown
Owner

@gustav-fff are you here?

@gustav-fff

Copy link
Copy Markdown
Collaborator

[triage-bot] DIRECTED: Here. Standing by for instructions.

Honk-Honk 🪿

@dmtrKovalenko

Copy link
Copy Markdown
Owner

@gustav-fff review this PR lol

@gustav-fff

Copy link
Copy Markdown
Collaborator

[triage-bot] DIRECTED: Review.

Cross-repo PR; cannot push. Findings below, ordered by impact.

Blocking (perf):

  • crates/fff-python/src/lib.rs — every search/grep entry point (search, glob, directory_search, mixed_search, grep, multi_grep) holds the GIL across the full Rust call. For a project whose CLAUDE.md states "absolutely HIGHEST performance level", this is wrong: any Python thread is blocked for the entire fuzzy/grep run, and Rust's internal rayon pool can't be fed from other Python threads. Wrap the heavy section in py.allow_threads(|| { ... }). Read locks on picker and query_tracker need to be acquired inside allow_threads (or dropped before reacquiring GIL) to keep Send-safe.

Should fix:

  • mixed_search (lib.rs:680-725) builds a full FileItem/DirItem then copies every field into MixedFileItem/MixedDirItem. Either drop the Mixed* duplicates and reuse FileItem/DirItem, or construct Mixed* directly from &fff::FileItem. Current code clones every string twice.
  • health_check returns a JSON string (lib.rs:1034). Un-Pythonic — caller has to json.loads. Build a PyDict directly. Doing so also lets you drop serde_json from Cargo.toml.
  • __exit__ (lib.rs:442) is a no-op while destroy exists. PR body advertises lifecycle via context manager; either call self.destroy() from __exit__ or remove destroy and rely on Drop. Right now with FileFinder(...) as f: does not deterministically release LMDB handles — it waits for refcount.
  • multi_grep accepts no mode parameter and silently forces PlainText (lib.rs:880). Either expose mode for parity with grep, or document the constraint in the docstring.
  • reindex (lib.rs:975) passes cache_budget: None, discarding the budget the user constructed in __new__. Persist the original budget on FileFinder and reuse it.

Nits:

  • lib.rs:493, 667 — combo_boost_score_multiplier: combo_boost_multiplier (clippy redundant_field_names-adjacent; rename param to match).
  • pyproject.toml Repository/Issues URLs point to dmtrKovalenko/fff.nvim; the actual repo is dmtrKovalenko/fff.
  • search/glob/etc. use page_size=0 as a sentinel for "100". Idiomatic Python would be page_size: Optional[int] = None. Not blocking but trips type-checkers.
  • tests/test_finder.py — no test covers track_query / get_historical_query / refresh_git_status / reindex / pagination. Consider one smoke test each.
  • The Score match_type: String (lib.rs:48) does an allocation per result; if MatchType is a small enum upstream, expose it as a Python str literal via interning or as an int enum.

LGTM:

  • Workspace + maturin setup is clean, abi3-py310 is the right call.
  • CI matrix covers ubuntu/macos/windows.
  • Result-type surface mirrors the Rust API faithfully.

Recommend addressing the GIL release and mixed_search double-clone before merge; the rest can be follow-up.

Honk-Honk 🪿

Comment thread packages/fff-python/README.md Outdated
Rust bindings:

- Release GIL during heavy search/grep operations via py.allow_threads

- Add MixedFileItem/MixedDirItem::from_core to avoid double-cloning

- Return PyDict directly from health_check and drop serde_json dependency

- Call destroy() in __exit__ so the context manager releases resources

- Add mode parameter to multi_grep for parity with grep

- Preserve cache budget overrides across reindex()

- Rename combo_boost param to match FuzzySearchOptions field

Docs/tests:

- Update Python test for dict-returning health_check

- Add Python bindings section to main README

CI:

- Fix pypi-publish job to depend on build-python/build-python-sdist

  instead of release, making the workflow_dispatch checkbox functional
@4fuu

4fuu commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

This is the logic I added for the release CI. Please take a look and let me know if it makes sense.


How to publish fff-python to PyPI

One-time setup

Add a Trusted Publisher in the PyPI project settings:

  • Owner: dmtrKovalenko
  • Repository: fff
  • Workflow: .github/workflows/release.yaml
  • Environment: pypi

To enable automatic publishing on tags, add a repository variable:

  • Settings → Secrets and variables → Actions → Variables
  • Name: PUBLISH_TO_PYPI
  • Value: true

Option 1: Publish automatically on tag push

Run the release script locally:

./scripts/release.sh 0.x.x

It automatically bumps the Python package version numbers and creates/pushes the release tag. The tag push triggers the release workflow, which builds all artifacts, creates a GitHub release, and publishes the Python wheels + sdist to PyPI when PUBLISH_TO_PYPI is true.

Option 2: Publish manually

  1. Go to Actions → Prebuild → Run workflow
  2. Select the branch
  3. Check publish_pypi
  4. Click Run workflow

This builds and uploads the Python wheels + sdist to PyPI, but does not create a GitHub release or bump any version numbers.

Notes

  • The publish_pypi checkbox only works for workflow_dispatch. Tag pushes use the PUBLISH_TO_PYPI variable instead.
  • Manual runs still build Neovim / C / MCP artifacts, but only Python gets published.
  • Pushes to main create a nightly GitHub release but do not publish to PyPI.

@4fuu

4fuu commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Hey, I went through the review points from you and gustav-fff and fixed all of them in the latest commit:

  • Released the GIL during heavy search/grep calls
  • Removed the double cloning in mixed_search
  • health_check now returns a Python dict directly (and dropped serde_json)
  • __exit__ actually calls destroy() now
  • multi_grep got a mode param
  • reindex() keeps the cache budget overrides
  • Fixed the combo boost naming inconsistency
  • Added Python bindings info to the main README
  • Fixed the release CI so the manual PyPI checkbox actually works

Also updated the Python test and pushed everything. Let me know if anything looks off!

4fuu and others added 5 commits June 16, 2026 13:15
- Split crates/fff-python/src/lib.rs into modules:
  - types.rs: all pyclass result types
  - finder.rs: FileFinder implementation
  - conversions.rs: From/core conversions

- Make API more Pythonic:
  - FileFinder now accepts pathlib.Path / os.PathLike for base_path and reindex
  - Add close() alias for destroy()
  - grep/multi_grep now raise FFFException for invalid modes
  - GrepResult gains has_more property and next_cursor() method

- Add type stubs:
  - packages/fff-python/src/fff/__init__.pyi
  - packages/fff-python/src/fff/py.typed

- Add __repr__ implementations for all exposed pyclasses

- Expand Python tests for pathlib, close(), reprs, invalid mode, and cursor pagination
Make the Python binding API more idiomatic before merge: replace getter-style methods with properties, keep close() as the single explicit shutdown API, add container semantics for result objects, and tighten type stubs with Literal/Sequence/PathLike support.

Also reduce Rust binding duplication with shared option/result helpers and release the Python GIL around blocking filesystem, git, and query-history operations.
@dmtrKovalenko

Copy link
Copy Markdown
Owner

made wait for scan api async and some additional changes

@dmtrKovalenko dmtrKovalenko merged commit 65832db into dmtrKovalenko:main Jun 17, 2026
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants