'chunk size' = 128MB and 256MB, with automatic chunking scheme not creating 128MB or 256MB of chunk size.

[tinaok]

I created a branch with 'debug' in my repo. There for each operation, it shows dataset information with chunk size. I created a table from output of debug information as follows.
https://github.com/tinaok/benchmarking/blob/debug/chunksize.md

For Automatic chunking scheme, I tried to re-calculate the chunking size, out of number of element each chunk size that each debug output showed.
For the benchmark with 512MB, chunk size =(251, 192, 160) : it reproduce 520 * 384 * 320 * 8 = 511MB so it is ok.
But for 256MB , chunk size =(317, 192, 160): it is 317 * 192 * 160 * 8=78MB, so chunking size is not working.
Same for 128MB, chunk size =(251, 192, 160): it is 251 * 192* 160 * 8=62MB, so chunking size is not working.

anyidia what's going on here?

[tinaok]

I made a little script for reproducing this problem.
https://gist.github.com/tinaok/c2ef193e94508a5ba426979d01e99307
from chunk_size 380MB, automatic chunk size creates '380MB' chunk size, but if chunk_size is less than 370MB, automatic chunk size does not create 370MB, it only creates less than half size of chunk size.....

[tinaok]

dask/dask#5439 (comment)
Hi @andersy005 & @kmpaul
as you can see from this issue, 'auto' does not create exact chunk size as it is defined in dask.config.set({"array.chunk-size": '256MB'}),
for example when setting 256MB, it created only 78MB
any thoughts on this?

[andersy005]

@tinaok, thank you for pointing this out! This is something worth looking into. I will follow the discussion in dask/dask#5439.