Merge upstream Vortex 0.76.0 into spiceai-54#71
Merged
Conversation
fixup links for vortex from spiral and the `docs.` subdomain Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ex-data#8374) <!-- Thank you for submitting a pull request! We appreciate your time and effort. Please make sure to provide enough information so that we can review your pull request. The Summary and Testing sections below contain guidance on what to include. --> ## Summary <!-- If this PR is related to a tracked effort, please link to the relevant issue here (e.g., `Closes: vortex-data#123`). Otherwise, feel free to ignore / delete this. In this section, please: 1. Explain the rationale for this change. 2. Summarize the changes included in this PR. A general rule of thumb is that larger PRs should have larger summaries. If there are a lot of changes, please help us review the code by explaining what was changed and why. If there is an issue or discussion attached, there is no need to duplicate all the details, but clarity is always preferred over brevity. --> This PR adds support for import/export to Arrow for the `Point` extension type, as the `geoarrow.point` Arrow extension with separated (struct) coordinates. Stacked on vortex-data#8372. <!-- ## API Changes Uncomment this section if there are any user-facing changes. Consider whether the change affects users in one of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the engine integrations. 3. Should some documentation be updated to reflect this change? If a public API is changed in a breaking manner, make sure to add the appropriate label. --> ## Testing <!-- Please describe how this change was tested. Here are some common categories for testing in Vortex: 1. Verifying existing behavior is maintained. 2. Verifying new behavior and functionality works correctly. 3. Serialization compatibility (backwards and forwards) should be maintained or explicitly broken. --> Unit tests are added to exercise both code paths, plus a Vortex → Arrow → Vortex round-trip. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Support `<REGEX>:` and `<!REGEX>:` in sqllogictests Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
We are no longer using the lockfiles, but I accidentally added this one as part of vortex-data#7722 Signed-off-by: Andrew Duffy <andrew@a10y.dev>
## Summary I was trying [hawkeye](https://github.com/korandoru/hawkeye) which has some nice features `reuse` doesn't have, and it found this weird issue. Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Adds some benchmarks for binary operators as I have some plans to improve Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
This PR adds 2 kernels for OnPair: - byte_length which uses uncompressed_lengths only - constant comparison to avoid decompressing for queries like URL != '' This PR also adds validity() for FSST's kernel for byte_length so that evaluating validity wouldn't mean decompressing whole FSST array. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | com.palantir.java-format | `2.91.0` → `2.93.0` |  |  | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
## Summary An experiment to align the packed buffers in the benchmarks to try and reduce some of the noise around the `compare` benchmark. If this works maybe its a path towards more consistent benchmarks? Signed-off-by: Adam Gutglick <adam@spiraldb.com>
## Summary Yet another iteration of our renovate config, including the following changes: 1. Removes some already-default base configs or moved things there. 2. Lock files updated weekly, everything else every second Monday 3. JS/NPM - only update the lockfile and security updates 4. The lance benchmarks have their own group, so they shouldn't impact other dependencies Signed-off-by: Adam Gutglick <adam@spiraldb.com>
## Summary Closes: vortex-data#8429 Split the lance-specific rule into two. Signed-off-by: Adam Gutglick <adam@spiraldb.com>
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [pip](https://redirect.github.com/pypa/pip) ([changelog](https://pip.pypa.io/en/stable/news/)) | `26.1.1` → `26.1.2` |  |  | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### [CVE-2026-8643](https://nvd.nist.gov/vuln/detail/CVE-2026-8643) / PYSEC-2026-196 <details> <summary>More information</summary> #### Details pip would treat console_scripts and gui_scripts as paths instead of file names without sanitizing the resolved absolute path to the installation directory, leading to entry points being installed outside the installation directory. #### Severity - CVSS Score: 5.5 / 10 (Medium) - Vector String: `CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:N` #### References - [http://www.openwall.com/lists/oss-security/2026/06/01/5](http://www.openwall.com/lists/oss-security/2026/06/01/5) - [https://mail.python.org/archives/list/security-announce@python.org/thread/YV63UET5D3OOJY7O4M5XCVYO2YM4NBYJ/](https://mail.python.org/archives/list/security-announce@python.org/thread/YV63UET5D3OOJY7O4M5XCVYO2YM4NBYJ/) - [https://github.com/pypa/pip/pull/14000](https://redirect.github.com/pypa/pip/pull/14000) This data is provided by [OSV](https://osv.dev/vulnerability/PYSEC-2026-196) and the [PyPI Advisory Database](https://redirect.github.com/pypa/advisory-database) ([CC-BY 4.0](https://redirect.github.com/pypa/advisory-database/blob/main/LICENSE)). </details> --- ### Release Notes <details> <summary>pypa/pip (pip)</summary> ### [`v26.1.2`](https://redirect.github.com/pypa/pip/compare/26.1.1...26.1.2) [Compare Source](https://redirect.github.com/pypa/pip/compare/26.1.1...26.1.2) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [duckdb](https://redirect.github.com/duckdb/duckdb-python) ([changelog](https://redirect.github.com/duckdb/duckdb-python/releases)) | `1.4.1` → `1.4.2` |  |  | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### [CVE-2025-64429](https://nvd.nist.gov/vuln/detail/CVE-2025-64429) / [GHSA-vmp8-hg63-v2hp](https://redirect.github.com/advisories/GHSA-vmp8-hg63-v2hp) / PYSEC-2025-112 <details> <summary>More information</summary> #### Details DuckDB is a SQL database management system. DuckDB implemented block-based encryption of DB on the filesystem starting with DuckDB 1.4.0. There are a few issues related to this implementation. The DuckDB can fall back to an insecure random number generator (pcg32) to generate cryptographic keys or IVs. When clearing keys from memory, the compiler may remove the memset() and leave sensitive data on the heap. By modifying the database header, an attacker could downgrade the encryption mode from GCM to CTR to bypass integrity checks. There may be a failure to check return value on call to OpenSSL `rand_bytes()`. An attacker could use public IVs to compromise the internal state of RNG and determine the randomly generated key used to encrypt temporary files, get access to cryptographic keys if they have access to process memory (e.g. through memory leak),circumvent GCM integrity checks, and/or influence the OpenSSL random number generator and DuckDB would not be able to detect a failure of the generator. Version 1.4.2 has disabled the insecure random number generator by no longer using the fallback to write to or create databases. Instead, DuckDB will now attempt to install and load the OpenSSL implementation in the `httpfs` extension. DuckDB now uses secure MbedTLS primitive to clear memory as recommended and requires explicit specification of ciphers without integrity checks like CTR on `ATTACH`. Additionally, DuckDB now checks the return code. #### Severity - CVSS Score: 6.5 / 10 (Medium) - Vector String: `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:N` #### References - [https://github.com/duckdb/duckdb/blob/029a5b87ff5b1cd22f7f9717d48cd8830d00807c/src/common/random_engine.cpp#L20](https://redirect.github.com/duckdb/duckdb/blob/029a5b87ff5b1cd22f7f9717d48cd8830d00807c/src/common/random_engine.cpp#L20) - [https://duckdb.org/2025/09/16/announcing-duckdb-140.html](https://duckdb.org/2025/09/16/announcing-duckdb-140.html) - [https://github.com/duckdb/duckdb/security/advisories/GHSA-vmp8-hg63-v2hp](https://redirect.github.com/duckdb/duckdb/security/advisories/GHSA-vmp8-hg63-v2hp) - [https://github.com/duckdb/duckdb/pull/17275](https://redirect.github.com/duckdb/duckdb/pull/17275) This data is provided by [OSV](https://osv.dev/vulnerability/PYSEC-2025-112) and the [PyPI Advisory Database](https://redirect.github.com/pypa/advisory-database) ([CC-BY 4.0](https://redirect.github.com/pypa/advisory-database/blob/main/LICENSE)). </details> --- ### Release Notes <details> <summary>duckdb/duckdb-python (duckdb)</summary> ### [`v1.4.2`](https://redirect.github.com/duckdb/duckdb-python/releases/tag/v1.4.2): Python DuckDB v1.4.2 [Compare Source](https://redirect.github.com/duckdb/duckdb-python/compare/v1.4.1...v1.4.2) This is a bug fix release for various issues discovered after we released v1.4.1. Also see the [DuckDB v1.4.2 Changelog](https://redirect.github.com/duckdb/duckdb/releases/tag/v1.4.2). #### What's Changed - Add Python 3.14 support by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;116](https://redirect.github.com/duckdb/duckdb-python/pull/116) - Fix ADBC driver path resolution when `importlib.util` was not implicitly loaded by [@&vortex-data#8203;henryharbeck](https://redirect.github.com/henryharbeck) in [#&vortex-data#8203;135](https://redirect.github.com/duckdb/duckdb-python/pull/135) - add targeted test workflow by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;145](https://redirect.github.com/duckdb/duckdb-python/pull/145) - Remove xfail annotations on adbc tests by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;147](https://redirect.github.com/duckdb/duckdb-python/pull/147) - fix config dict value typehint by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;151](https://redirect.github.com/duckdb/duckdb-python/pull/151) - Add df data and tz type columns back into the same loc after type con… by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;150](https://redirect.github.com/duckdb/duckdb-python/pull/150) - Enable pyarrow with python 3.14 by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;152](https://redirect.github.com/duckdb/duckdb-python/pull/152) - spark imports by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;157](https://redirect.github.com/duckdb/duckdb-python/pull/157) - Fix failing test due to changed error msg by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;158](https://redirect.github.com/duckdb/duckdb-python/pull/158) - Add explicit .pl(lazy=True) overload by [@&vortex-data#8203;J-Meyers](https://redirect.github.com/J-Meyers) in [#&vortex-data#8203;172](https://redirect.github.com/duckdb/duckdb-python/pull/172) - Fix InsertRelation on attached database by [@&vortex-data#8203;evertlammerts](https://redirect.github.com/evertlammerts) in [#&vortex-data#8203;155](https://redirect.github.com/duckdb/duckdb-python/pull/155) **Full Changelog**: <duckdb/duckdb-python@v1.4.1...v1.4.2> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [anthropics/claude-code-action](https://redirect.github.com/anthropics/claude-code-action) ([changelog](https://redirect.github.com/anthropics/claude-code-action/compare/fbda2eb1bdc90d319b8d853f5deb53bca199a7c1..d5726de019ec4498aa667642bc3a80fca83aa102)) | action | digest | `fbda2eb` → `d5726de` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, on the second Monday and fourth Monday of the month (`* 0-3 * * 1#2,1#4`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [gradle/actions](https://redirect.github.com/gradle/actions) ([changelog](https://redirect.github.com/gradle/actions/compare/50e97c2cd7a37755bbfafc9c5b7cafaece252f6e..3f131e8634966bd73d06cc69884922b02e6faf92)) | action | digest | `50e97c2` → `3f131e8` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, on the second Monday and fourth Monday of the month (`* 0-3 * * 1#2,1#4`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [taiki-e/install-action](https://redirect.github.com/taiki-e/install-action) ([changelog](https://redirect.github.com/taiki-e/install-action/compare/0631aa6515c7d545823c67cfae7ef4fc7f490154..7a79fe8c3a13344501c80d99cae481c1c9085912)) | action | digest | `0631aa6` → `7a79fe8` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, on the second Monday and fourth Monday of the month (`* 0-3 * * 1#2,1#4`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMTkuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIxOS4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Match rust API: buffer is borrowed under the condition it won't be modified until data source is freed Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Spark mostly focuses on sizeInBytes which we populate from file sizes with scaling. We also report numRows since that exists in our datasource. Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary Closes: #000 <!-- ## API Changes Uncomment this section if there are any user-facing changes. Consider whether the change affects users in one of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the engine integrations. 3. Should some documentation be updated to reflect this change? If a public API is changed in a breaking manner, make sure to add the appropriate label. --> ## Testing <!-- Please describe how this change was tested. Here are some common categories for testing in Vortex: 1. Verifying existing behavior is maintained. 2. Verifying new behavior and functionality works correctly. 3. Serialization compatibility (backwards and forwards) should be maintained or explicitly broken. --> --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
See https://github.com/vortex-data/vortex/actions/runs/27608329337/job/81625965956?pr=8121, This job hangs Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary Previously `Mask::execute` accepted **nullable** boolean arrays and silently coerced null elements to `false` (folding validity into the value bits). That behaviour was flagged with a `TODO` questioning whether the implicit coercion was correct. This PR makes `Mask` execution **strict**: `Mask::execute` now requires a non-nullable boolean array (`Bool(NonNullable)`) and errors on nullable input. The null-as-`false` coercion is no longer implicit — callers that want SQL-style `NULL`-as-not-matching semantics opt in explicitly via `array.fill_null(false)` before executing into a `Mask`. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude <noreply@anthropic.com>
CUDA san is slow gate it on CUDA changes. This is will post commit with san --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…x-data#8436) ## Summary Split out of vortex-data#8425 so each perf change can be looked at on its own, as suggested by @joseph-isaacs. This is the "bit collect" half, no count fusion. `bitwise_binary_op` only took the direct word path when both operands were 64-bit aligned, otherwise falling back to `iter_padded`. It now reads the backing bytes as `u64` via `as_chunks::<8>` for any byte-aligned offset, leaving `iter_padded` for sub-byte offsets only. This also drops the `from_trusted_len_iter` unsafe for a safe `BufferMut` push. --------- Signed-off-by: Han Damin <miniex@daminstudio.net> Co-authored-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary This PR introduces a new `Validity::definitely_all_null()` method that provides a fast path to check if a validity is definitively all-null (i.e., `Validity::AllInvalid`). This is the counterpart to the existing `definitely_no_nulls()` method. This is a rename, but not a change in semantics --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The only remaining calls to Arrow are for kleene logic, arithmetic, and LIKE expressions. If we make these native Vortex, we can pull out Arrow into a vortex-arrow engine/integration crate. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com> Signed-off-by: "Nicholas Gates" <nick@nickgates.com>
Flag to benchmark correlated or uniform vectors separately. Useful for building flamegraphs with samply Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Add a private metadata-only bridge on PyVortex arrays so the optional CUDA extension can reconstruct arrays through its own local Vortex session instead of passing Rust ArrayRef values across Python extension modules. The CUDA extension now parses the metadata tree, calls local array plugins for deserialization, and exports bufferless arrays through the Arrow C Device capsule path. Physical buffer handoff remains a follow-up. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ex-data#8614) ## Summary `id()` methods for encodings, scalar/aggregate functions, layouts and extension dtypes built their `Id` via `Id::new`/`Id::new_static`, which interns into a globally locked table (`ThreadedRodeo`) on every call. As noted in vortex-data#8380 this made identifier construction a hot-path bottleneck under multi-threaded and random access. They now read a `CachedId` static that interns once on first access and returns the cached `Id` lock-free thereafter; the interned value is unchanged, so encoding and serialization behaviour is identical. This is the migration half of vortex-data#8380; the workspace `clippy` lint that forbids the locking constructors (so the regression can't return) follows in a separate PR. Refs: vortex-data#8380 ## Testing `cargo nextest run -p vortex-session -p vortex-array -p vortex-layout -p vortex-geo -p vortex-tensor -p vortex-json` passes; `cargo +nightly fmt --all --check` and `cargo clippy --all-targets` clean. `vortex-cuda` could not be built locally (no CUDA toolchain); its one migrated `id()` uses the same pattern. --- I'm Korean, so sorry if any wording reads a little awkward. Signed-off-by: Han Damin <miniex@daminstudio.net>
Stop using deprecated api
Support of bool -> Primitive casts. This is a subtask for pushing expressions of form "WHERE prefix(...) > 0" to Duckdb. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Polish treemap view in vortex explorer Signed-off-by: Robert Kruszewski <github@robertk.io>
Decode sliced bit-packed arrays in padded coordinates by sizing and launching for offset + len. This keeps the returned offset..offset+len device slice in bounds and ensures the final touched 1024-value chunk is decoded. Signed-off-by: "Alexander Droste" <alexander.droste@protonmail.com>
## Rationale for this change I want to move array buffers between devices, but keep the same array tree. Signed-off-by: "Nicholas Gates" <nick@nickgates.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> --------- Signed-off-by: Robert Kruszewski <github@robertk.io> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Robert Kruszewski <github@robertk.io>
CI coverage for the Python CUDA bridge through the vortex-data[cuda] optional-extra path. Extend the private metadata bridge to carry host buffer-export capsules instead of only a buffer count. The base Python package exports repr(C) VortexBufferExport descriptors, and vortex-python-cuda imports them into local BufferHandles before deserializing arrays through its own VortexSession. Tests now cover primitive, nullable, bool, and struct arrays across the bridge, plus the existing CUDA Arrow Device smoke path. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ng array construction (vortex-data#8626) Patches::cast_values was a crutch and sign that in too many places we were producing wrong dtypes. This pr removes this function and fixes up dtypes where necessary. We still need a cast in ALP-RD take as ALP-RD patches must be non nullable
…data#8617) ## Summary Follow-up to vortex-data#8614. A `clippy` disallowed-methods lint now forbids `Id::new` and `Id::new_static` workspace-wide, since both intern into a globally locked table (`ThreadedRodeo`) on every call. Static identifiers must use a `CachedId` static instead; the sites that intern a genuinely dynamic string (proto/flatbuffer deserialization, arrow extension lookups, Python read contexts, tests) keep `Id::new` behind an `#[expect(clippy::disallowed_methods, reason = "...")]`. `CachedId` itself now interns through `Id::new_static`, so it is the single sanctioned path for static ids. This is the enforcement half of vortex-data#8380; vortex-data#8614 did the migration. It is stacked on vortex-data#8614, so it stays a draft until that merges; the shared migration commit drops out once this is rebased onto a merged vortex-data#8614. Refs: vortex-data#8380 ## Testing `cargo clippy --all-targets` is clean across the edited crates (`vortex-array`, `vortex-layout`, `vortex-file`, `vortex-ipc`, `vortex-zstd`, `vortex-python`, `vortex-duckdb`, `vortex-session`), with `disallowed_methods` and `unfulfilled_lint_expectations` both at zero; `cargo +nightly fmt --all --check` clean. `vortex-cuda` could not be built locally (no CUDA toolchain). --- I'm Korean, so sorry if any wording reads a little awkward. Signed-off-by: Han Damin <miniex@daminstudio.net>
Previously, when printing Id in GDB, thread have been changing constantly and stack frame was lost. This PR resolves that. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…#8630) ## Rationale for this change Running the benchmark harness (`datafusion-bench` / `query_bench`) against a remote dataset on Google Cloud Storage (`--opt remote-data-dir=gs://…`) fails immediately during benchmark setup with: ``` Error: unknown URL scheme: gs ``` `vortex-bench`'s `url_scheme_to_storage` helper — which maps a data-dir URL scheme to a `storage` label used for result reporting — only handled `s3` and `file`, so any `gs://` run bailed before a single query executed. S3 remote runs work because `s3` is handled; GCS was simply never covered. `make_object_store` already supports `gs://` for the actual reads, so the only gap was this reporting helper. ## What changes are included in this PR? - Add a `STORAGE_GCS = "gcs"` constant. - Add a `"gs"` arm to `url_scheme_to_storage` returning that label. Verified by running TPC-H SF1 from a GCS bucket end-to-end (DataFusion + Vortex, 22/22 queries executing against `gs://…`, results tagged `storage=gcs`). ## What APIs are changed? Are there any user-facing changes? None. This only affects the benchmark harness's storage-label reporting; no public API, format, or behavior change outside `vortex-bench`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rtex-data#8598) <!-- Thank you for submitting a pull request! We appreciate your time and effort. Please make sure to provide enough information so that we can review your pull request. The Summary and Testing sections below contain guidance on what to include. --> ## Summary <!-- If this PR is related to a tracked effort, please link to the relevant issue here (e.g., `Closes: vortex-data#123`). Otherwise, feel free to ignore / delete this. In this section, please: 1. Explain the rationale for this change. 2. Summarize the changes included in this PR. A general rule of thumb is that larger PRs should have larger summaries. If there are a lot of changes, please help us review the code by explaining what was changed and why. If there is an issue or discussion attached, there is no need to duplicate all the details, but clarity is always preferred over brevity. --> Adds a SpatialBench benchmark that runs on DuckDB across two storage lanes, Vortex binary BLOB and Parquet binary BLOB. <!-- ## API Changes Uncomment this section if there are any user-facing changes. Consider whether the change affects users in one of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the engine integrations. 3. Should some documentation be updated to reflect this change? If a public API is changed in a breaking manner, make sure to add the appropriate label. --> ## Benchmark results <!-- Please describe how this change was tested. Here are some common categories for testing in Vortex: 1. Verifying existing behavior is maintained. 2. Verifying new behavior and functionality works correctly. 3. Serialization compatibility (backwards and forwards) should be maintained or explicitly broken. --> Local, warm, median of 5 (ms). | Q | Query | Parquet | Vortex | Winner | |---|---|--:|--:|:--| | Q1 | trips within 50km of Sedona | 170.2 ms | 132.7 ms | Vortex 1.3× | | Q2 | trips in Coconino County | 379.5 ms | 249.2 ms | Vortex 1.5× | | Q3 | monthly stats within 15km | 235.5 ms | 203.3 ms | Vortex 1.2× | | Q4 | top-1000-tip zone distribution | 404.8 ms | 114.8 ms | Vortex 3.5× | | Q5 | repeat-customer convex hulls | 3174.8 ms | 3187.3 ms | tie | | Q6 | zone stats in bbox | 718.8 ms | 306.1 ms | Vortex 2.3× | | Q7 | route-detour ratios | 938.6 ms | 1046.1 ms | tie (~noise) | | Q8 | nearby pickups per building (500m) | 1000.6 ms | 1071.2 ms | tie (~noise) | | Q9 | building conflation (IoU) | 36.4 ms | 40.6 ms | tie | | Q10 | zone stats (LEFT JOIN) | timeout | timeout | — | | Q11 | cross-zone trips | timeout | timeout | — | | Q12 | 5 nearest buildings (KNN) | timeout | timeout | — | ## Takeaways Vortex wins the scan/filter-bound queries (Q1–Q4, Q6) by up to 3.5×. Q5/Q7/Q8/Q9 are compute-bound (DuckDB spatial join dominates), so the formats converge. Q10–Q12 time out on both, because the unindexed spatial join is the wall, independent of storage format. --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…on management (vortex-data#8631) I think immutable session is nice but hard to integrate it with for a lot of systems
Before this PR, we were converting Vortex extension dtypes e.g. timestamp to underlying types i.e. i64 instead of TIMESTAMP while creating a Duckdb Value. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…lumns in files (vortex-data#8634) This code mostly existed before. Swimlane is a new top level tab alongside current view as it replicates the schema of the table to render the lanes
Upgrade the Spice.ai Vortex fork from 0.75.0 to 0.76.0 (161 upstream commits). Conflict resolution: - vortex-session: took upstream's session.rs refactor (VortexSession moved to a submodule with a copy-on-write ArcSwap model, which subsumes the Spice.ai get() deadlock fix); re-added the Spice.ai set() convenience method. - vortex-array executor.rs + optimizer/kernels.rs: upstream reimplemented the per-ExecutionCtx execute-parent kernel snapshot natively (ParentExecutionKernels), superseding the Spice.ai KernelSnapshot patch. Took upstream and removed the now-orphaned ArcSwapMap::load_full. - vortex-datafusion sink.rs/format.rs/exprs.rs/lib.rs: kept the Spice.ai VortexSink direct-write + target_file_size feature and the DefaultExpressionConvertor / simple-CASE / predicate-skip enhancements, integrated alongside upstream's DisplayAs file_groups improvement and case-when upstreaming. Fixed unused_qualifications and clippy absolute_paths lints newly enforced in 0.76.0. - vortex-file/vortex-layout scan sub-split: the feature was upstreamed into 0.76.0, so took upstream tests.rs (adapted to the new assert_arrays_eq! ctx argument). - Scoped #[allow(deprecated)] for the extension date->timestamp cast and the Arrow Map helper, which run through APIs (CastReduce / execute_arrow) that expose no ExecutionCtx to migrate to the new ctx-based execute::<T>(ctx) path. Verified: cargo check --workspace (excl. vortex-duckdb/vortex-python), cargo clippy -D warnings, cargo +nightly fmt --check, and cargo nextest (3498 passed, 1 skipped) across the touched crates. Signed-off-by: Luke Kim <luke@spice.ai>
…to ctx execute APIs Replace the deprecated ctx-free helpers (to_primitive, to_extension, to_canonical, scalar_at, Validity::to_mask) and the deprecated ArrowArrayExecutor::execute_arrow with the 0.76 ctx-based execute::<T>(ctx) / ArrowSession APIs, removing the #[allow(deprecated)] shims added during the 0.76.0 merge. - Split the extension cast to match how Primitive splits reduce vs kernel: CastReduce keeps only buffer-free restructuring (and drops the spurious to_canonical on the same-ext path), while the Date->Timestamp value conversion moves to a new CastKernel for Extension (registered via CastExecuteAdaptor(Extension)), where an ExecutionCtx is available. Behavior is preserved: different-ext casts still error, and the date->timestamp conversion (null handling plus the exactness/overflow checks) is unchanged. - Read per-row validity via Validity::execute_mask once instead of per-element scalar_at. - The Arrow Map round-trip test converts back via session.arrow().execute_arrow(...). Verified: cargo clippy -D warnings and cargo nextest run -p vortex-array (3015 passed, 1 skipped). Signed-off-by: Luke Kim <luke@spice.ai>
krinart
approved these changes
Jul 6, 2026
pull Bot
pushed a commit
to TheRakeshPurohit/spiceai
that referenced
this pull request
Jul 7, 2026
* chore(deps): upgrade Vortex to 0.76.0 Bump the vendored spiceai/vortex crates from the spiceai-54 tip (ab4f0b17) to 409505df, which merges upstream Vortex 0.76.0 into the spiceai-54 branch (spiceai/vortex#71). - Cargo.toml: update the vortex / vortex-array / vortex-btrblocks / vortex-scan / vortex-session / vortex-utils git rev in both [workspace.dependencies] and [patch.crates-io], annotating the label with the version (vortex tags publish 0.1.0, so the rev is the only version signal). - Cargo.lock: the vortex crates move to the new rev and vortex-compute is new in 0.76.0. Re-resolving the 0.76.0 graph also re-unifies a few already-present transitive deps (windows-sys / itertools / socket2 / windows-core edges); no resolved package versions change and the lock is settled (cargo metadata --locked passes). - crates/vortex (vendored vortex-datafusion): 0.76.0 removed vortex::expr::pruning::checked_pruning_expr — the pruning submodule moved out of vortex-array::expr into vortex-file. Port the one affected test to the new Expression::falsify(scope, session) API. Validated locally: vortex-datafusion and cayenne (the only crates that use Vortex APIs) compile, test, and lint clean across lib + tests. * test(vortex): assert IN-list pruning references id min/max stats Strengthen test_in_list_conversion_produces_pruning_expression per PR review: assert the falsification output contains stat($.id, vortex.min()) and stat($.id, vortex.max()) rather than merely containing "id", so the test fails if the converted IN-list ever stops being derived from min/max statistics. * fix(deps): bump crossbeam-epoch to 0.9.20 (RUSTSEC-2026-0204) cargo-deny advisories was failing CI: crossbeam-epoch 0.9.18 (pulled via clickhouse-rs) is affected by RUSTSEC-2026-0204 (invalid pointer deref in the fmt::Pointer impl); 0.9.20 is the fixed release. Patch-level transitive bump only (version + checksum), no API impact. Pre-existing on trunk and unrelated to the Vortex upgrade — surfaced by an advisory-DB refresh.
github-actions Bot
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jul 8, 2026
* chore(deps): upgrade Vortex to 0.76.0 Bump the vendored spiceai/vortex crates from the spiceai-54 tip (ab4f0b17) to 409505df, which merges upstream Vortex 0.76.0 into the spiceai-54 branch (spiceai/vortex#71). - Cargo.toml: update the vortex / vortex-array / vortex-btrblocks / vortex-scan / vortex-session / vortex-utils git rev in both [workspace.dependencies] and [patch.crates-io], annotating the label with the version (vortex tags publish 0.1.0, so the rev is the only version signal). - Cargo.lock: the vortex crates move to the new rev and vortex-compute is new in 0.76.0. Re-resolving the 0.76.0 graph also re-unifies a few already-present transitive deps (windows-sys / itertools / socket2 / windows-core edges); no resolved package versions change and the lock is settled (cargo metadata --locked passes). - crates/vortex (vendored vortex-datafusion): 0.76.0 removed vortex::expr::pruning::checked_pruning_expr — the pruning submodule moved out of vortex-array::expr into vortex-file. Port the one affected test to the new Expression::falsify(scope, session) API. Validated locally: vortex-datafusion and cayenne (the only crates that use Vortex APIs) compile, test, and lint clean across lib + tests. * test(vortex): assert IN-list pruning references id min/max stats Strengthen test_in_list_conversion_produces_pruning_expression per PR review: assert the falsification output contains stat($.id, vortex.min()) and stat($.id, vortex.max()) rather than merely containing "id", so the test fails if the converted IN-list ever stops being derived from min/max statistics. * fix(deps): bump crossbeam-epoch to 0.9.20 (RUSTSEC-2026-0204) cargo-deny advisories was failing CI: crossbeam-epoch 0.9.18 (pulled via clickhouse-rs) is affected by RUSTSEC-2026-0204 (invalid pointer deref in the fmt::Pointer impl); 0.9.20 is the fixed release. Patch-level transitive bump only (version + checksum), no API impact. Pre-existing on trunk and unrelated to the Vortex upgrade — surfaced by an advisory-DB refresh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades the Spice.ai Vortex fork from 0.75.0 → 0.76.0 (161 upstream commits) via the merge-based patches-branch workflow, then migrates the remaining Spice.ai feature code off APIs that 0.76.0 deprecated.
Two commits:
Merge tag '0.76.0' into lukim/spiceai-54-vortex-0.76.0refactor(vortex-array): migrate extension date-cast + Arrow Map test to ctx execute APIsConflict resolution
Several Spice.ai patches were upstreamed into 0.76.0, so upstream's (more thorough) version was taken rather than carrying redundant code:
vortex-sessionsession.rsrefactor (copy-on-writeArcSwapmodel, which subsumes the Spice.aiget()deadlock fix); re-added the Spice.aiset()convenience method.vortex-arrayexecutor +optimizer/kernels.rsExecutionCtxkernel snapshot natively (ParentExecutionKernels), superseding the Spice.aiKernelSnapshotpatch → took upstream; removed the now-orphanedArcSwapMap::load_full.vortex-datafusionsink/format/exprs/libtarget_file_sizeand theDefaultExpressionConvertor/ simple-CASE/ predicate-skip enhancements, integrated with upstream'sDisplayAsimprovement + case-when upstreaming. Fixedunused_qualificationsand clippyabsolute_paths(newly enforced).vortex-file/vortex-layoutscan sub-splittests.rs(adapted to the newassert_arrays_eq!(…, &mut ctx)).Preserved Spice.ai patches: VortexSink
target_file_size,WriteStrategyBuildersession support, the extension date→timestamp cast, and the Arrow Map alias.Deprecated-API migration (commit 2)
0.76.0 deprecates the ctx-free canonicalization helpers (
to_primitive,to_extension,to_canonical,scalar_at,Validity::to_mask) andArrowArrayExecutor::execute_arrow. Because CI builds with-D warnings, these had to be removed (not just#[allow(deprecated)]-shimmed):Primitivedoes it.CastReducekeeps only buffer-free restructuring (and drops a spuriousto_canonicalon the same-ext path); the Date→Timestamp value conversion moves to a newCastKernel for Extension(registered viaCastExecuteAdaptor(Extension)), where anExecutionCtxis available. Behavior is preserved — different-ext casts still error, and the date→timestamp conversion (null handling + exactness/overflow checks) is unchanged.Validity::execute_maskonce instead of per-elementscalar_at.session.arrow().execute_arrow(...).No
#[allow(deprecated)]remains in the fork's patch code.Verification
cargo check --workspace(excl.vortex-duckdb/vortex-python— external toolchains) — 0 errorscargo clippy --all-targets -- -D warningson the touched crates — cleancargo +nightly fmt --all --check— cleancargo nextest runonvortex-{session,array,layout,file,datafusion}— 3498 passed, 1 skipped; fullvortex-arraysuite 3015 passedCargo.lockis--locked-consistentNot run locally (need external toolchains / left to CI):
vortex-duckdb(DUCKDB_SOURCE_DIR),vortex-python(pyo3), and the Java bindings.DataFusion stays at 54 (no major bump this release).