feat: add rustler_precompiled support for prebuilt NIF binaries #70
Conversation
Replace source compilation with prebuilt NIF downloads so users no longer need Rust, cmake, or pkg-config installed. First compile drops from 5-10 minutes to seconds. Changes: - mix.exs: add rustler_precompiled dep, make rustler optional, include checksum files in hex package - native.ex: use RustlerPrecompiled with 6 Linux/macOS targets - Cargo.toml: pin NIF version 2.15 feature, add release LTO profile - .cargo/config.toml: cross-compilation flags for macOS and musl - release.yml: GitHub Actions workflow to build NIFs for all targets on tag push using philss/rustler-precompiled-action Users can force local compilation with ECTO_LIBSQL_BUILD=true.
WalkthroughThis pull request introduces precompiled NIF support for the ecto_libsql project. A GitHub Actions workflow builds and releases precompiled NIFs across multiple platforms. The NIF loading mechanism is updated from static Rustler to RustlerPrecompiled with dynamic configuration. Rust and Elixir build tooling are updated accordingly with optimisation flags and feature configurations. Changes
Sequence DiagramsequenceDiagram
participant GHA as GitHub Actions
participant Build as Build Process<br/>(Cargo)
participant Artifacts as Artifact Storage<br/>(GitHub Releases)
participant Runtime as Runtime<br/>(BEAM VM)
GHA->>Build: Trigger build for each<br/>target/OS/NIF combination
Build->>Build: Compile native code<br/>with rustflags & LTO
Build->>Artifacts: Upload precompiled<br/>NIF binaries
Artifacts->>Artifacts: Publish archives on<br/>version tag
Runtime->>Runtime: Load RustlerPrecompiled
Runtime->>Runtime: Read project version<br/>& GitHub URL
Runtime->>Artifacts: Fetch precompiled NIF<br/>for target platform
Artifacts->>Runtime: Deliver NIF binary
Runtime->>Runtime: Load NIF into BEAM
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~50 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In @.github/workflows/release.yml:
- Around line 51-61: The workflow references a non-existent action version for
philss/rustler-precompiled-action in the Build the project step (id:
build-crate); update the action tag in that step from `@v1.1.4` to the published
`@v1.1.3` so GitHub Actions can resolve the dependency (change the uses entry for
philss/rustler-precompiled-action accordingly).
In `@mix.exs`:
- Around line 70-71: The package.files list in mix.exs (the files: ~w(...)
tuple) is missing docs referenced in docs/0 (specifically
ECTO_MIGRATION_GUIDE.md and SECURITY.md); update the files entry in mix.exs to
include those two filenames so they are shipped to Hex (add
"ECTO_MIGRATION_GUIDE.md" and "SECURITY.md" to the ~w(...) list alongside
README.md, LICENSE, CHANGELOG.md, USAGE.md, etc.).
- Around line 6-8: Update the Rustler dependency in mix.exs so it's always
optional (replace the conditional `optional: not (`@dev`? or `@force_build`?)` with
`optional: true` for the `{:rustler, "~> 0.37.1", ...}` tuple), and stop using
dependency optionality to control builds—use `config :rustler_precompiled,
force_build: ...` in your config instead; also add "ECTO_MIGRATION_GUIDE.md" and
"SECURITY.md" to the package files list (the files array in mix.exs) so they are
included in the distributed package.
In `@native/ecto_libsql/Cargo.toml`:
- Line 19: Update the rustler dependency version in Cargo.toml: change the
rustler entry that currently reads version "0.37.0" to version "0.37.1" so it
matches the Elixir mix.exs constraint (~> 0.37.1) and pulls the retired 0.37.1
fixes for rustler/rustler_mix; locate the rustler = { ... } dependency line and
update its version field accordingly.
🧹 Nitpick comments (3)
Cargo.toml (1)
7-8: Consider usinglto = "thin"instead oflto = truefor better build-time trade-offs.LTO is compatible with cdylib/Rustler NIFs and is recommended in Rustler's official precompilation guidance. However,
lto = true(fat LTO) incurs substantial link-time cost. Current Rust guidance suggestslto = "thin"as a practical compromise—it achieves nearly the same runtime performance with significantly reduced build time..github/workflows/release.yml (2)
17-20: Unused permissions declared.
id-token: writeandattestations: writeare declared but not used by any step in this workflow. If these are intended for future attestation features, consider adding a comment; otherwise, removing them follows the principle of least privilege.🔒 Proposed fix to remove unused permissions
permissions: contents: write - id-token: write - attestations: write
42-46: Add validation for extracted version.The
sedextraction silently produces an empty string if the pattern doesn't match (e.g., indentation changes). Subsequent steps would receive an invalid version, potentially causing confusing build failures.🛡️ Proposed fix to validate version extraction
- name: Extract project version shell: bash run: | # Extract version from mix.exs `@version` attribute - echo "PROJECT_VERSION=$(sed -n 's/^ `@version` "\(.*\)"/\1/p' mix.exs | head -n1)" >> $GITHUB_ENV + VERSION=$(sed -n 's/^[[:space:]]*@version "\(.*\)"/\1/p' mix.exs | head -n1) + if [ -z "$VERSION" ]; then + echo "::error::Failed to extract version from mix.exs" + exit 1 + fi + echo "PROJECT_VERSION=$VERSION" >> $GITHUB_ENV
| @dev? String.ends_with?(@version, "-dev") | ||
| @force_build? System.get_env("ECTO_LIBSQL_BUILD") in ["1", "true"] | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -e "mix.exs" --max-one-resultRepository: ocean/ecto_libsql
Length of output: 294
🏁 Script executed:
head -100 mix.exsRepository: ocean/ecto_libsql
Length of output: 2798
🏁 Script executed:
sed -n '55,75p' mix.exsRepository: ocean/ecto_libsql
Length of output: 824
🌐 Web query:
RustlerPrecompiled optional Rustler dependency missing artefacts force build
💡 Result:
If RustlerPrecompiled can’t find/download a precompiled NIF for your platform (missing artefacts), the supported fallback is to force a local build via Rustler—but that only works if Rustler is present as a dependency.
1) Add :rustler as an (optional) dependency
defp deps do
[
{:rustler_precompiled, "~> 0.8"},
{:rustler, ">= 0.0.0", optional: true}
]
endRustlerPrecompiled explicitly requires this to be able to force-build. [1][2]
2) Force-build for your OTP app
In config/config.exs (or runtime.exs if you want it environment-driven):
config :rustler_precompiled, :force_build, your_otp_app: trueThis forces Rustler compilation instead of downloading artefacts. [1][2]
3) (Optional) Force-build for all RustlerPrecompiled packages
config :rustler_precompiled, force_build_all: trueor set:
RUSTLER_PRECOMPILED_FORCE_BUILD_ALL=true[3]
If you share your use RustlerPrecompiled, otp_app: ..., crate: ... stanza and the exact “missing artefacts” error, I can tell you whether it’s an unsupported target vs. missing published assets/checksum (which are different fixes). [4]
Sources: [1] [2] [3] [4]
🌐 Web query:
RustlerPrecompiled hex.pm documentation optional dependency
💡 Result:
In rustler_precompiled, Rustler is an optional dependency (so most users don’t need Rust installed unless they fall back to building locally) [1], and Hex lists it as rustler ~> 0.23 (optional) [2].
If you want to force building from source (instead of downloading a precompiled NIF), the docs say you must add Rustler explicitly to your app’s deps as an optional dependency:
{:rustler, ">= 0.0.0", optional: true}This is documented in RustlerPrecompiled’s Hex docs and also appears in the library source error message shown when a download fails [3] [4].
Sources:
[1] https://hexdocs.pm/rustler_precompiled/changelog.html
[2] https://hex.pm/packages/rustler_precompiled
[3] https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html
[4] https://preview.hex.pm/preview/rustler_precompiled/show/lib/rustler_precompiled.ex
Rustler optionality should be unconditional for RustlerPrecompiled fallback to work.
RustlerPrecompiled expects Rustler to be available as an optional dependency so it can fall back to local compilation when precompiled artefacts are unavailable. The current conditional pattern—optional: not (@dev? or @force_build?)—breaks this fallback for users on unsupported platforms who aren't developing locally and haven't set ECTO_LIBSQL_BUILD.
Change line 61 to make Rustler always optional:
{:rustler, "~> 0.37.1", optional: true},Use configuration (config :rustler_precompiled, force_build: ...) instead of dependency optionality to control build behaviour in development and CI environments.
Additionally, add "ECTO_MIGRATION_GUIDE.md" and "SECURITY.md" to the package files list (lines 67–68); both are documented as extras but missing from the distributed package.
🤖 Prompt for AI Agents
In `@mix.exs` around lines 6 - 8, Update the Rustler dependency in mix.exs so it's
always optional (replace the conditional `optional: not (`@dev`? or
`@force_build`?)` with `optional: true` for the `{:rustler, "~> 0.37.1", ...}`
tuple), and stop using dependency optionality to control builds—use `config
:rustler_precompiled, force_build: ...` in your config instead; also add
"ECTO_MIGRATION_GUIDE.md" and "SECURITY.md" to the package files list (the files
array in mix.exs) so they are included in the distributed package.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Fetch the release page content to understand what the user is pointing to
gh release view v1.1.4 --repo philss/rustler-precompiled-action --json name,tagName,body,url
Length of output: 498
🌐 Web query:
rustler_precompiled conditional optional dependency pattern dev force_build
💡 Result:
rustler_precompiled is designed so end users don’t need Rust (it downloads precompiled NIFs). You only need the :rustler dependency when you want to force a local build (e.g., in dev, CI, or when precompiled artifacts aren’t available). The docs explicitly say you must add Rustler as a dependency (typically optional) to enable :force_build. [1][2]
Recommended “conditional optional dependency” pattern
mix.exs
defp deps do
[
{:rustler_precompiled, "~> 0.8"},
# Only needed when you want to compile from source (force_build)
{:rustler, ">= 0.0.0", optional: true, runtime: false}
]
endoptional: true keeps installs fast for users who only download precompiled artifacts, while still allowing builds-from-source when enabled. [1][2]
Force build only in dev (or via env var)
config/dev.exs
config :rustler_precompiled, :force_build, my_otp_app: trueThis switches rustler_precompiled into “build from source via Rustler” mode for that OTP app. [1][2]
Force build automatically for “-dev” / pre-release versions
If you set your NIF/package :version to a pre-release like "0.1.0-dev", rustler_precompiled treats :force_build as enabled (still requiring :rustler to be present as a dependency). [2][3]
“Force build everything” (all deps using rustler_precompiled)
You can force builds globally with either:
config :rustler_precompiled, force_build_all: true, or- env var
RUSTLER_PRECOMPILED_FORCE_BUILD_ALL[2]
Sources: RustlerPrecompiled docs and implementation notes on :force_build, :force_build_all, and requiring {:rustler, ">= 0.0.0", optional: true}. [1][2][3]
@ricardo-valero, I apologize—I misread the conditional logic. Your pattern is actually correct!
optional: not (@dev? or @force_build?) means:
- For end users (production, no
ECTO_LIBSQL_BUILD): Rustler is optional ✓ - For development (
-devversion): Rustler is required (ensures force_build works) - When explicitly building (
ECTO_LIBSQL_BUILDset): Rustler is required
This aligns with rustler_precompiled best practices: Rustler is optional for end users (allowing fallback compilation if precompiled artifacts are missing), while ensuring it's available when you need to force local builds during development or CI.
The conditional dependency pattern is a valid approach here. Thank you for the context with the action release link—v1.1.4 includes the fixes that ensure the workflow builds correctly.
One remaining suggestion: consider adding "ECTO_MIGRATION_GUIDE.md" and "SECURITY.md" to your package files list (line 68), since they're listed as :extras in docs but won't be distributed with the Hex package currently.
🧠 Learnings used
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to mix.exs : Update version number during release process and ensure Hex package includes `lib/`, `priv/`, `native/`, and documentation files while excluding `test/`, `examples/`, and build artifacts
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to **/*.{ex,exs} : Always run formatters before committing: `mix format && cd native/ecto_libsql && cargo fmt`, then verify checks pass with `mix format --check-formatted && cargo fmt --check` before executing `git commit`
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to lib/ecto/adapters/libsql/connection.ex : SQL generation for queries and DDL operations, as well as expression and constraint handling, should be implemented in this module
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to lib/ecto/adapters/libsql.ex : Storage operations, type loaders/dumpers, and migration logic should be implemented in this module
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to native/ecto_libsql/src/**/*.rs : All Rust NIF functions must be decorated with `#[rustler::nif(schedule = "DirtyIo")]` and return `NifResult<T>` instead of using `.unwrap()` which can panic the BEAM VM
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to **/*.rs : Always run Rust formatter before committing: `cargo fmt` at native/ecto_libsql, then verify checks pass with `cargo fmt --check` before executing `git commit`
Learnt from: CR
Repo: ocean/ecto_libsql PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-18T06:10:14.791Z
Learning: Applies to lib/ecto_libsql/native.ex : Provide safe wrapper functions in EctoLibSql.Native that handle NIF stub definitions and error conversion from Rust NIF returns
- Revert .gitignore changes (not part of this PR) - Bump Cargo.toml rustler from 0.37.0 to 0.37.1 (0.37.0 was retired) - Add ECTO_MIGRATION_GUIDE.md and SECURITY.md to hex package files - Use lto = "thin" instead of "true" for better build-time trade-off - Remove unused id-token and attestations permissions from release.yml - Add version extraction validation to fail fast on parse errors
1 participant
Summary
rustler_precompiledsupport so users download prebuilt NIF binaries instead of compiling 244 Rust crate dependencies from sourceECTO_LIBSQL_BUILD=trueChanges
Elixir (
mix.exs,lib/ecto_libsql/native.ex){:rustler_precompiled, "~> 0.8"}as a runtime dependency{:rustler, "~> 0.37.1"}optional (only required for-devversions or whenECTO_LIBSQL_BUILD=true)use Rustlerwithuse RustlerPrecompiledinEctoLibSql.Nativewith:aarch64-apple-darwin,x86_64-apple-darwin,aarch64-unknown-linux-gnu,x86_64-unknown-linux-gnu,aarch64-unknown-linux-musl,x86_64-unknown-linux-musl2.15(OTP 22+, forward-compatible with 2.16/2.17)checksum-*.exsto hex package files list@dev?and@force_build?module attributes for build controlRust (
Cargo.toml,.cargo/config.toml)nif_version_2_15feature withdefault-features = false[profile.release]with LTO at workspace root (Cargo requires profiles at workspace level).cargo/config.tomlwith cross-compilation flags:-C link-arg=-undefined -C link-arg=dynamic_lookup(dynamic NIF symbol resolution)-C target-feature=-crt-static(BEAM dynamic loader compatibility)CI (
.github/workflows/release.yml)philss/rustler-precompiled-action@v1.1.4v*), native code changes to main, workflow_dispatchsoftprops/action-gh-release@v2on tagscrossfor Linux ARM64 and musl targetsTarget Platforms
aarch64-apple-darwinmacos-14x86_64-apple-darwinmacos-13aarch64-unknown-linux-gnuubuntu-22.04x86_64-unknown-linux-gnuubuntu-22.04aarch64-unknown-linux-muslubuntu-22.04x86_64-unknown-linux-muslubuntu-22.04Windows targets excluded for now —
libsql-ffiC/C++ bindgen+cmake makes Windows cross-compilation complex. Can be added in a follow-up.Release Process (after merge)
mix.exs+Cargo.tomlgit tag v0.X.0 && git push origin main --tagsmix rustler_precompiled.download EctoLibSql.Native --all --printTesting
cargo test)mix test)mix format --check-formatted)ECTO_LIBSQL_BUILD=trueReferences
Summary by CodeRabbit
New Features
Chores
✏️ Tip: You can customize this high-level summary in your review settings.