chore: Adopt uv as package manager#3608
Conversation
|
sure, it's probably about time i learned to use uv properly 😄 |
FBruzzesi
changed the title
|
Random version failure is unrelated (GHA) See failed tests
=========================== short test summary info ============================
FAILED tests/expr_and_series/concat_str_test.py::test_concat_str_with_large_string - pyarrow.lib.ArrowNotImplementedError: Function 'binary_join_element_wise' has no kernel matching input types (large_string, string, large_string)
FAILED tests/frame/join_test.py::test_join_on_null_values[polars[eager]-anti-expected5] - ValueError: zip() argument 2 is longer than argument 1
FAILED tests/frame/join_test.py::test_join_on_null_values[polars[lazy]-anti-expected5] - ValueError: zip() argument 2 is longer than argument 1
FAILED tests/frame/join_test.py::test_join_on_null_values[polars[lazy]-semi-expected4] - ValueError: zip() argument 2 is shorter than argument 1
FAILED tests/frame/join_test.py::test_join_on_null_values[polars[eager]-semi-expected4] - ValueError: zip() argument 2 is shorter than argument 1
= 5 failed, 7406 passed, 633 skipped, 306 xfailed, 2 xpassed in 62.39s (0:01:02) = |
This comment was marked as resolved.
This comment was marked as resolved.
| (?x) | ||
| ^(tests/utils\.py) | ||
| |^(test_plugin/) | ||
| |^(packages/test_plugin/) |
There was a problem hiding this comment.
I'm commenting here since it's the only change in .pre-commit-config.yaml
Has anyone brought up the autoupdate feature as an issue?
narwhals/.pre-commit-config.yaml
Lines 1 to 6 in c0f674a
I know that ibis uses renovate, which seems to integrate with lock files.
Wondering if we need to change anything here?
There was a problem hiding this comment.
not sure what you're asking here sorry
There was a problem hiding this comment.
Sorry, short version is this @MarcoGorelli
Do we need to change anything related to pre-commit auto update?
dangotbanned
left a comment
dangotbanned
left a comment
There was a problem hiding this comment.
@FBruzzesi this huge huge HUGE effort looks good great to me.
I've left a few more comments, but nothing is blocking so feed me please
camriddell
left a comment
camriddell
left a comment
There was a problem hiding this comment.
Thank you for all of your work @FBruzzesi this is shaping up really well. I have a few questions & comments that should be addressed on:
- Changes made to existing dependency versions in pyproject.toml
- Documentation for running the test suite
- Changes that appear unrelated to the goal of the PR (adopting UV)
|
|
||
| 4. Activate it. On Linux, this is `. .venv/bin/activate`, on Windows `.\.venv\Scripts\activate`. | ||
| ```terminal | ||
| uv sync --group local-dev --extra dask --extra pyspark --extra modin |
There was a problem hiding this comment.
By default uv sync is "exact" meaning repeated calls remove extraneous existing packages to ensure the environment ONLY has the packages specified:
❯ uv sync --group local-dev --extra modin
Resolved 167 packages in 1ms
Installed 3 packages in 9ms
+ fsspec==2026.4.0
+ modin==0.37.1
+ psutil==7.2.2
❯ uv sync --group local-dev --extra dask
Resolved 167 packages in 1ms
Uninstalled 2 packages in 4ms
Installed 7 packages in 10ms
+ cloudpickle==3.1.2
+ dask==2026.3.0
+ importlib-metadata==9.0.0
+ locket==1.0.0
- modin==0.37.1 # modin was removed!
+ partd==1.4.2
- psutil==7.2.2
+ toolz==1.1.0
+ zipp==3.23.1To avoid this behavior we can push users towards just specifying packages in uv run ...
uv run --group local-dev --extra modin pytest --constructors modinAlternatively we could mention that one can use the uv sync --inexact ... flag to avoid the accidental removal of existing packages.
❯ uv sync --group local-dev --extra dask
Resolved 167 packages in 1ms
Installed 6 packages in 7ms
+ cloudpickle==3.1.2
+ dask==2026.3.0
+ fsspec==2026.4.0
+ locket==1.0.0
+ partd==1.4.2
+ toolz==1.1.0
❯ uv sync --inexact --group local-dev --extra modin
Resolved 167 packages in 2ms
Uninstalled 1 package in 12ms
Installed 5 packages in 45ms
+ modin==0.37.1
- pandas==3.0.3
+ pandas==2.3.3
+ psutil==7.2.2
+ pytz==2026.2
+ tzdata==2026.2There was a problem hiding this comment.
Was this tested after adding the conflicts into pyproject.toml? LMK if you also get this output or if I did something funny here.
❯ uv sync --group local-dev --extra dask --extra pyspark --extra modin
Using CPython 3.14.2
Creating virtual environment at: .venv
Resolved 167 packages in 2ms
error: Extras `dask` and `modin` are incompatible with the declared conflicts: {`narwhals[cudf]`, `narwhals[dask]`, `narwhals[duckdb]`, `narwhals[modin]`, `narwhals:extreme-minimum-versions`, `narwhals:extreme-pretty-old-versions`}
| "black>=26.3.1", # required by mkdocstrings_handlers | ||
| "jinja2>=3.1.6", | ||
| "markdown-exec[ansi]>=1.12.1", | ||
| "mkdocs-autorefs>=1.4.4", |
There was a problem hiding this comment.
why do these packages have minimum version pins now?
There was a problem hiding this comment.
i'd also prefer to keep orthogonal changes to a separate pr, especially if a pr starts touching 30 files or so
MarcoGorelli
left a comment
MarcoGorelli
left a comment
There was a problem hiding this comment.
thanks for working on this
tbh i'm not totally sold on the dependency management, as show dependencies doesn't necessarily match what gets run when you add in --extra, e.g.
(narwhals-dev) mgorelli@marcoslaptop:~/narwhals-dev$ uv tree --group core-tests | grep pandas
Resolved 167 packages in 3ms
│ ├── pandas v3.0.3 (extra: dataframe)
│ ├── pandas v3.0.3 (extra: connect) (*)
│ ├── pandas v3.0.3 (group: core-tests) (*)
├── pandas v3.0.3 (group: core-tests) (*)
(narwhals-dev) mgorelli@marcoslaptop:~/narwhals-dev$ uv run --group core-tests --extra modin python -c 'import pandas; print(pandas.__version__)'
2.3.3So, i'm a little hesitant about replacing the github workflows (the rest of the changes look fine though)
Thank @MarcoGorelli that was the reason why I was using uv pip compile, which allows to pass extras other than groups. The case here is modin forcing pandas to be below v3. I will keep looking for a way of doing this |
|
@MarcoGorelli Need to take care of some pre-release deps in the lock file. Not sure I can make it today. |
|
Hey everyone, thanks for all the reviews and iterations you are doing. I think I addressed all the open threads, but there are quite a few so I might have missed some. There are quite a few follow ups (Thanks @dangotbanned for tracking them all in different issues). Some comments to address:
Let me know if there is something else to address |
5 participants
Description
Related discord thread
Closes #881 as
build-backend = "uv_build"requires that4.7k over 5.1k lines are due to committing the uv.lock file - The main reason for that is that these days security vulnerabilities are a nightmare and we could get some automatic dependabot alerts if we have a lock file. This is the same reason for which I started to pin some non-core dependencies (e.g. pytest has vulnerabilities fixed in v9.0.3).
uv auditis still experimental but also a good start to run periodically locally to check for known vulnerabilitiesWhat type of PR is this? (check all applicable)