[BUG] write zarr error

[shanzha9]

Describe the bug
Description

Calling adata.write_zarr() fails when adata.X is a cupy-backed dask array that has been processed with:

sc.pp.scale(adata, zero_center=True)

The error occurs during da.store() inside anndata._io.specs.methods.write_basic_dask_dask_dense.

The failure seems related to missing dask graph dependencies generated by the scaling kernel.

Error
ValueError: Missing dependency
('scale_kernel-da48f1f4eeff8dc05cd605e935e53c7f', 187, 0)
for dependents
{('store-map-cddb4ee2da9b05dc96e2bd8042893736', 187, 0)}
Steps/Code to reproduce bug
Follow this guide http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.

Expected behavior

Name Version Build Channel

_openmp_mutex 4.5 20_gnu conda-forge
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.13.5 pypi_0 pypi
aiosignal 1.4.0 pypi_0 pypi
anndata 0.12.13 pypi_0 pypi
array-api-compat 1.14.0 pypi_0 pypi
asttokens 3.0.1 pypi_0 pypi
attrs 26.1.0 pypi_0 pypi
bokeh 3.9.0 pypi_0 pypi
bzip2 1.0.8 hda65f42_9 conda-forge
ca-certificates 2026.4.22 hbd8a1cb_0 conda-forge
cachetools 7.1.1 pypi_0 pypi
certifi 2026.4.22 pypi_0 pypi
click 8.3.3 pypi_0 pypi
cloudpickle 3.1.2 pypi_0 pypi
comm 0.2.3 pypi_0 pypi
contourpy 1.3.3 pypi_0 pypi
cuda-bindings 12.9.6 pypi_0 pypi
cuda-core 0.7.0 pypi_0 pypi
cuda-pathfinder 1.5.4 pypi_0 pypi
cuda-python 12.9.6 pypi_0 pypi
cuda-toolkit 12.9.2.0 pypi_0 pypi
cudf-cu12 26.4.0 pypi_0 pypi
cugraph-cu12 26.4.0 pypi_0 pypi
cuml-cu12 26.4.0 pypi_0 pypi
cupy-cuda12x 14.0.1 pypi_0 pypi
cuvs-cu12 26.4.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dask 2026.1.1 pypi_0 pypi
dask-cuda 26.4.0 pypi_0 pypi
dask-cudf-cu12 26.4.0 pypi_0 pypi
debugpy 1.8.20 pypi_0 pypi
decorator 5.2.1 pypi_0 pypi
distributed 2026.1.1 pypi_0 pypi
distributed-ucxx-cu12 0.49.0 pypi_0 pypi
docrep 0.3.2 pypi_0 pypi
donfig 0.8.1.post1 pypi_0 pypi
et-xmlfile 2.0.0 pypi_0 pypi
executing 2.2.1 pypi_0 pypi
fast-array-utils 1.4.1 pypi_0 pypi
fonttools 4.62.1 pypi_0 pypi
frozenlist 1.8.0 pypi_0 pypi
fsspec 2026.4.0 pypi_0 pypi
google-crc32c 1.8.0 pypi_0 pypi
h5py 3.16.0 pypi_0 pypi
icu 78.3 h33c6efd_0 conda-forge
idna 3.13 pypi_0 pypi
imageio 2.37.3 pypi_0 pypi
ipykernel 7.2.0 pypi_0 pypi
ipython 9.13.0 pypi_0 pypi
ipython-pygments-lexers 1.1.1 pypi_0 pypi
ipywidgets 8.1.8 pypi_0 pypi
jedi 0.20.0 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
joblib 1.5.3 pypi_0 pypi
jupyter-client 8.8.0 pypi_0 pypi
jupyter-core 5.9.1 pypi_0 pypi
jupyterlab-widgets 3.0.16 pypi_0 pypi
kiwisolver 1.5.0 pypi_0 pypi
lazy-loader 0.5 pypi_0 pypi
ld_impl_linux-64 2.45.1 default_hbd61a6d_102 conda-forge
legacy-api-wrap 1.5 pypi_0 pypi
libcudf-cu12 26.4.0 pypi_0 pypi
libcugraph-cu12 26.4.0 pypi_0 pypi
libcuml-cu12 26.4.0 pypi_0 pypi
libcuvs-cu12 26.4.0 pypi_0 pypi
libexpat 2.8.0 hecca717_0 conda-forge
libffi 3.5.2 h3435931_0 conda-forge
libgcc 15.2.0 he0feb66_18 conda-forge
libgomp 15.2.0 he0feb66_18 conda-forge
libkvikio-cu12 26.4.0 pypi_0 pypi
liblzma 5.8.3 hb03c661_0 conda-forge
libmpdec 4.0.0 hb03c661_1 conda-forge
libraft-cu12 26.4.0 pypi_0 pypi
librmm-cu12 26.4.0 pypi_0 pypi
libsqlite 3.53.0 hf4e2dac_0 conda-forge
libstdcxx 15.2.0 h934c35e_18 conda-forge
libucx-cu12 1.19.0 pypi_0 pypi
libucxx-cu12 0.49.0 pypi_0 pypi
libuuid 2.42 h5347b49_0 conda-forge
libzlib 1.3.2 h25fd6f3_2 conda-forge
llvmlite 0.46.0 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
markdown-it-py 4.0.0 pypi_0 pypi
markupsafe 3.0.3 pypi_0 pypi
matplotlib 3.10.9 pypi_0 pypi
matplotlib-inline 0.2.1 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
msgpack 1.1.2 pypi_0 pypi
multidict 6.7.1 pypi_0 pypi
narwhals 2.20.0 pypi_0 pypi
natsort 8.4.0 pypi_0 pypi
ncurses 6.6 hdb14827_0 conda-forge
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.6.1 pypi_0 pypi
numba 0.64.0 pypi_0 pypi
numba-cuda 0.28.2 pypi_0 pypi
numcodecs 0.16.5 pypi_0 pypi
numpy 2.4.4 pypi_0 pypi
nvidia-cublas-cu12 12.9.2.10 pypi_0 pypi
nvidia-cuda-cccl-cu12 12.9.27 pypi_0 pypi
nvidia-cuda-nvcc-cu12 12.9.86 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.9.86 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.9.79 pypi_0 pypi
nvidia-cufft-cu12 11.4.1.4 pypi_0 pypi
nvidia-cufile-cu12 1.14.1.1 pypi_0 pypi
nvidia-curand-cu12 10.3.10.19 pypi_0 pypi
nvidia-cusolver-cu12 11.7.5.82 pypi_0 pypi
nvidia-cusparse-cu12 12.5.10.65 pypi_0 pypi
nvidia-libnvcomp-cu12 5.1.0.21 pypi_0 pypi
nvidia-ml-py 13.595.45 pypi_0 pypi
nvidia-nccl-cu12 2.30.4 pypi_0 pypi
nvidia-nvfatbin-cu12 12.9.82 pypi_0 pypi
nvidia-nvjitlink-cu12 12.9.86 pypi_0 pypi
nvtx 0.2.15 pypi_0 pypi
openpyxl 3.1.5 pypi_0 pypi
openssl 3.6.2 h35e630c_0 conda-forge
packaging 26.2 pypi_0 pypi
pandas 2.3.3 pypi_0 pypi
parso 0.8.7 pypi_0 pypi
partd 1.4.2 pypi_0 pypi
patsy 1.0.2 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pillow 12.2.0 pypi_0 pypi
pip 26.1.1 pyh145f28c_0 conda-forge
platformdirs 4.9.6 pypi_0 pypi
prompt-toolkit 3.0.52 pypi_0 pypi
propcache 0.4.1 pypi_0 pypi
psutil 7.2.2 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.3 pypi_0 pypi
pyarrow 24.0.0 pypi_0 pypi
pygments 2.20.0 pypi_0 pypi
pylibcudf-cu12 26.4.0 pypi_0 pypi
pylibcugraph-cu12 26.4.0 pypi_0 pypi
pylibraft-cu12 26.4.0 pypi_0 pypi
pynndescent 0.6.0 pypi_0 pypi
pyparsing 3.3.2 pypi_0 pypi
python 3.13.13 h6add32d_100_cp313 conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python_abi 3.13 8_cp313 conda-forge
pytz 2026.2 pypi_0 pypi
pyyaml 6.0.3 pypi_0 pypi
pyzmq 27.1.0 pypi_0 pypi
raft-dask-cu12 26.4.0 pypi_0 pypi
rapids-dask-dependency 26.4.0 pypi_0 pypi
rapids-logger 0.2.3 pypi_0 pypi
rapids-singlecell-cu12 0.15.0 pypi_0 pypi
readline 8.3 h853b02a_0 conda-forge
rich 15.0.0 pypi_0 pypi
rmm-cu12 26.4.0 pypi_0 pypi
scanpy 1.12.1 pypi_0 pypi
scikit-image 0.26.0 pypi_0 pypi
scikit-learn 1.8.0 pypi_0 pypi
scikit-misc 0.5.2 pypi_0 pypi
scipy 1.17.1 pypi_0 pypi
scverse-misc 0.0.5 pypi_0 pypi
seaborn 0.13.2 pypi_0 pypi
session-info2 0.4.1 pypi_0 pypi
six 1.17.0 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
stack-data 0.6.3 pypi_0 pypi
statsmodels 0.14.6 pypi_0 pypi
tblib 3.2.2 pypi_0 pypi
threadpoolctl 3.6.0 pypi_0 pypi
tifffile 2026.5.2 pypi_0 pypi
tk 8.6.13 noxft_h366c992_103 conda-forge
toolz 1.1.0 pypi_0 pypi
tornado 6.5.5 pypi_0 pypi
tqdm 4.67.3 pypi_0 pypi
traitlets 5.15.0 pypi_0 pypi
treelite 4.7.0 pypi_0 pypi
typing-extensions 4.15.0 pypi_0 pypi
tzdata 2026.2 pypi_0 pypi
ucxx-cu12 0.49.0 pypi_0 pypi
umap-learn 0.5.12 pypi_0 pypi
urllib3 2.6.3 pypi_0 pypi
wcwidth 0.7.0 pypi_0 pypi
widgetsnbextension 4.0.15 pypi_0 pypi
xyzservices 2026.3.0 pypi_0 pypi
yarl 1.23.0 pypi_0 pypi
zarr 3.2.0 pypi_0 pypi
zict 3.0.0 pypi_0 pypi
zstd 1.5.7 hb78ec9c_6 conda-forge

`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[34], line 1
----> 1 adata.write_zarr("/home/wangyufei-p/data/singlecell/cohort/1.annotation/scanpy_framework/test/aging_10M_checkpointa.zarr")

[... skipping hidden 1 frame]

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_core/anndata.py:2075, in AnnData.write_zarr(self, store, chunks, convert_strings_to_categoricals)
2070 msg = (
2071 "Passing write_zarr(adata, chunks=True) is no longer supported. "
2072 "Please pass write_zarr(adata) instead."
2073 )
2074 raise ValueError(msg)
-> 2075 write_zarr(
2076 store,
2077 self,
2078 chunks=chunks,
2079 convert_strings_to_categoricals=convert_strings_to_categoricals,
2080 )

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/utils.py:349, in no_write_dataset_2d..raise_error_if_dataset_2d_present(store, adata, *args, **kwargs)
342 msg = (
343 "Writing AnnData objects with a Dataset2D not supported yet. "
344 "Please use ds.to_memory to bring the dataset into memory. "
345 "Note that if you have generated this object by concatenating several AnnData objects"
346 "the original types may be lost."
347 )
348 raise NotImplementedError(msg)
--> 349 return write(store, adata, *args, **kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/zarr.py:59, in write_zarr(store, adata, chunks, convert_strings_to_categoricals, **ds_kwargs)
56 dataset_kwargs = dict(dataset_kwargs, chunks=chunks)
57 write_func(store, elem_name, elem, dataset_kwargs=dataset_kwargs)
---> 59 write_dispatched(f, "/", adata, callback=callback, dataset_kwargs=ds_kwargs)
60 if is_zarr_v2():
61 zarr.convenience.consolidate_metadata(f.store)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/experimental/_dispatch_io.py:74, in write_dispatched(store, key, elem, callback, dataset_kwargs)
70 from anndata._io.specs import _REGISTRY, Writer
72 writer = Writer(_REGISTRY, callback=callback)
---> 74 writer.write_elem(store, key, elem, dataset_kwargs=dataset_kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/utils.py:272, in report_write_key_on_error..func_wrapper(*args, **kwargs)
270 raise ValueError(msg)
271 try:
--> 272 return func(*args, **kwargs)
273 except Exception as e:
274 path = _get_display_path(store)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:390, in Writer.write_elem(self, store, k, elem, dataset_kwargs, modifiers)
388 if self.callback is None:
389 return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
--> 390 return self.callback(
391 write_func,
392 store,
393 k,
394 elem,
395 dataset_kwargs=dataset_kwargs,
396 iospec=self.registry.get_spec(elem),
397 )

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/zarr.py:57, in write_zarr..callback(write_func, store, elem_name, elem, dataset_kwargs, iospec)
51 if (
52 chunks is not None
53 and not isinstance(elem, sparse.spmatrix)
54 and elem_name.lstrip("/") == "X"
55 ):
56 dataset_kwargs = dict(dataset_kwargs, chunks=chunks)
---> 57 write_func(store, elem_name, elem, dataset_kwargs=dataset_kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:77, in write_spec..decorator..wrapper(g, k, *args, **kwargs)
75 @wraps(func)
76 def wrapper(g: GroupStorageType, k: str, *args, **kwargs):
---> 77 result = func(g, k, *args, **kwargs)
78 g[k].attrs.setdefault("encoding-type", spec.encoding_type)
79 g[k].attrs.setdefault("encoding-version", spec.encoding_version)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/methods.py:297, in write_anndata(f, k, adata, _writer, dataset_kwargs)
295 g = f.require_group(k)
296 if adata.X is not None:
--> 297 _writer.write_elem(g, "X", adata.X, dataset_kwargs=dataset_kwargs)
298 _writer.write_elem(g, "obs", adata.obs, dataset_kwargs=dataset_kwargs)
299 _writer.write_elem(g, "var", adata.var, dataset_kwargs=dataset_kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/utils.py:272, in report_write_key_on_error..func_wrapper(*args, **kwargs)
270 raise ValueError(msg)
271 try:
--> 272 return func(*args, **kwargs)
273 except Exception as e:
274 path = _get_display_path(store)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:390, in Writer.write_elem(self, store, k, elem, dataset_kwargs, modifiers)
388 if self.callback is None:
389 return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
--> 390 return self.callback(
391 write_func,
392 store,
393 k,
394 elem,
395 dataset_kwargs=dataset_kwargs,
396 iospec=self.registry.get_spec(elem),
397 )

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/zarr.py:57, in write_zarr..callback(write_func, store, elem_name, elem, dataset_kwargs, iospec)
51 if (
52 chunks is not None
53 and not isinstance(elem, sparse.spmatrix)
54 and elem_name.lstrip("/") == "X"
55 ):
56 dataset_kwargs = dict(dataset_kwargs, chunks=chunks)
---> 57 write_func(store, elem_name, elem, dataset_kwargs=dataset_kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:77, in write_spec..decorator..wrapper(g, k, *args, **kwargs)
75 @wraps(func)
76 def wrapper(g: GroupStorageType, k: str, *args, **kwargs):
---> 77 result = func(g, k, *args, **kwargs)
78 g[k].attrs.setdefault("encoding-type", spec.encoding_type)
79 g[k].attrs.setdefault("encoding-version", spec.encoding_version)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/methods.py:829, in write_cupy_dask(f, k, elem, _writer, dataset_kwargs)
828 def write_cupy_dask(f, k, elem, _writer, dataset_kwargs=MappingProxyType({})):
--> 829 _writer.write_elem(
830 f,
831 k,
832 elem.map_blocks(lambda x: x.get(), dtype=elem.dtype, meta=elem._meta.get()),
833 dataset_kwargs=dataset_kwargs,
834 )

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/utils.py:272, in report_write_key_on_error..func_wrapper(*args, **kwargs)
270 raise ValueError(msg)
271 try:
--> 272 return func(*args, **kwargs)
273 except Exception as e:
274 path = _get_display_path(store)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:390, in Writer.write_elem(self, store, k, elem, dataset_kwargs, modifiers)
388 if self.callback is None:
389 return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
--> 390 return self.callback(
391 write_func,
392 store,
393 k,
394 elem,
395 dataset_kwargs=dataset_kwargs,
396 iospec=self.registry.get_spec(elem),
397 )

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/zarr.py:57, in write_zarr..callback(write_func, store, elem_name, elem, dataset_kwargs, iospec)
51 if (
52 chunks is not None
53 and not isinstance(elem, sparse.spmatrix)
54 and elem_name.lstrip("/") == "X"
55 ):
56 dataset_kwargs = dict(dataset_kwargs, chunks=chunks)
---> 57 write_func(store, elem_name, elem, dataset_kwargs=dataset_kwargs)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/registry.py:77, in write_spec..decorator..wrapper(g, k, *args, **kwargs)
75 @wraps(func)
76 def wrapper(g: GroupStorageType, k: str, *args, **kwargs):
---> 77 result = func(g, k, *args, **kwargs)
78 g[k].attrs.setdefault("encoding-type", spec.encoding_type)
79 g[k].attrs.setdefault("encoding-version", spec.encoding_version)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/anndata/_io/specs/methods.py:537, in write_basic_dask_dask_dense(f, k, elem, _writer, dataset_kwargs)
535 else:
536 g = f.require_array(k, shape=elem.shape, dtype=elem.dtype, **dataset_kwargs)
--> 537 da.store(elem, g, scheduler="threads")

File /data/.conda/envs/ucx/lib/python3.13/site-packages/dask/array/core.py:1218, in store(failed resolving arguments)
1215 if not return_stored:
1216 import dask
-> 1218 dask.compute(arrays, **kwargs)
1219 return None
1220 else:

File /data/.conda/envs/ucx/lib/python3.13/site-packages/dask/base.py:685, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
682 expr = expr.optimize()
683 keys = list(flatten(expr.dask_keys()))
--> 685 results = schedule(expr, keys, **kwargs)
687 return repack(results)

File /data/.conda/envs/ucx/lib/python3.13/site-packages/dask/local.py:191, in start_state_from_dask(dsk, cache, sortkey, keys)
189 if task is None:
190 if dependents[key] and not cache.get(key):
--> 191 raise ValueError(
192 f"Missing dependency {key} for dependents {dependents[key]}"
193 )
194 continue
195 elif isinstance(task, DataNode):

ValueError: Missing dependency ('scale_kernel-da48f1f4eeff8dc05cd605e935e53c7f', 187, 0) for dependents {('store-map-cddb4ee2da9b05dc96e2bd8042893736', 187, 0)}
Error raised while writing key 'X' of <class 'zarr.core.group.Group'> to /`

[Intron7]

This is very likely you running out of memory. Can you confirm your dataset and chunk size as well as your gpu?

[shanzha9]

This is very likely you running out of memory. Can you confirm your dataset and chunk size as well as your gpu?

Thanks. I did not run out of memory. I have 4 X H200 and 1 TB RAM to deal with 10.1M cells. The chunk size is 20000. i can save the zarr before scaled, the error will rasie atfer scale and runpca. The nemory is enough.
# Example: use 4 local GPUs cluster = LocalCUDACluster( CUDA_VISIBLE_DEVICES="0,1,2,3", protocol="ucx", threads_per_worker=10, rmm_pool_size="8GB", rmm_maximum_pool_size="180GB", # 单卡留一点余量 rmm_allocator_external_lib_list=["cupy"], local_directory="/data/singlecell/cohort/1.annotation/scanpy_framework/test/", silence_logs="ERROR" ) client = Client(cluster); i run adata.X = adata.X.persist() wait(adata.X); after every step. adata.shape is (10269442, 2809) , adata.raw.shape is (10269442, 35288). if i did not saved zarr and go ahead, every line goes well ans the umap can be drawn.

Please help solve this question, thanks a lot. I can also mail you the jupyter.

The usage:

[shanzha9]

@Intron7 plz

[Intron7]

Update after digging in: this isn't an rsc bug, but you handed me the actual root cause clues in your reply (the LocalCUDACluster setup).

When a LocalCUDACluster client is active, .persist() parks the array's tasks as Future objects on the cluster workers. anndata.write_zarr then calls da.store(elem, g, scheduler="threads") — hardcoded threads scheduler — and the local threaded scheduler can't see Futures held by the distributed cluster, so it reports them as missing dependencies. This reproduces in six lines with no rsc, anndata, cupy, or zarr-specific logic in the picture:

from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import dask.array as da
import numpy as np
import zarr

cluster = LocalCUDACluster()
client = Client(cluster)

x = da.from_array(np.ones((10000, 200)), chunks=(1000, 200))
x = x.map_blocks(lambda b: b + 1).persist()

g = zarr.open("/tmp/test.zarr", mode="w", shape=x.shape, dtype=x.dtype, chunks=(1000, 200))
da.store(x, g, scheduler="threads")  # Missing dependency

Workaround: either adata.X = adata.X.compute() before adata.write_zarr(...) (materializes locally and bypasses the cluster Futures), or don't .persist() when a LocalCUDACluster client is active. .compute() is the most reliable option for your case.

I'll file the bug upstream on rapidsai/dask-cuda and link the issue here, then close this as not-rsc.

Two small notes for future reports:

• Please fill out the "Steps/Code to reproduce" section of the bug template. This one took noticeably longer to triage than necessary because there was no minimal reproducer; I burned a lot of time bisecting before discovering the cluster was the trigger — information that was in your second comment but would have been faster to find in a 10-line repro.
• On follow-up timing: your followup landed late evening my time and the @Intron7 plz ping arrived before I started my next workday. Please give at least a 2 full business day for the maintainer to respond before pinging.

[Intron7]

scverse/anndata#2444
rapidsai/dask-cuda#1649