Outstanding concept, difficult to start using

[hjalmarlucius]

The main reason I want to use nestedtensor and what value I want it to add

• Building a time series modelling system with pytorch where each step may be different.

The features I wish nestedtensor had

• Simpler install or build instructions
• (question) Can the tensor be mutated over time? I plan on adding incrementally more timesteps to the same tensor (like a list) - but from the outside it should look and feel like a tensor (of length t) at all times.

The things about nestedtensor that frustrate me

• I run py 3.9 and cuda 11.3 and there's no nightly binaries matching that spec. It might be obvious how but not to me.

[Optional] Example code or project I want to integrate with nestedtensor

[cpuhrsch]

Hello @hjalmarlucius, thank you for opening this issue and your interested in the project!

I can add binaries for the python and cuda version you're asking for. Indeed there's currently no nightlies that match that spec.

Currently the tensors cannot be mutated over time via something like "append". But you could use "cat" or "stack" to expand a NestedTensor (with the obvious performance penalty of memory allocation / deallocation). Also, for full transparency, note that it currently does not support autograd.

[okhat]

Thanks for the awesome work! Any update here about binaries for cuda 11 (on any python 3.7+)?

Our ColBERT work and its many derivatives in IR and NLP will benefit dramatically from this!

[cpuhrsch]

Hello @okhat, thank you for your interest! I've unfortunately not had the chance to work on this. Do you need autograd for your project?

[okhat]

@cpuhrsch I don't need autograd! I just need fast inference with variable-sized matrics.

I'm trying to install on Linux + cuda 11 and to my understanding there's only support for macosx? https://download.pytorch.org/whl/nightly/cu111/torch_nightly.html

FYI: so far we've rolled our own nested tensor implementation around torch.as_strided. But it's neither convenient nor as fast as we'd like :-)

[okhat]

Would it be easy to build manually using setup.py? I'm happy to try that — would appreciate if you could know if there are specific dependencies (or other instructions) to keep in mind!

[cpuhrsch]

@okhat - installing from source would indeed be easier for now. I'd recommend the following command

NVCC_FLAGS="-arch=sm_80" DEBUG=0 USE_NINJA=1 FORCE_CUDA=1 pip install -v -e .

You don't need any dependencies other than a recent nightly Keep in mind that NestedTensor operations need to be executed with the inference_mode context.

The CI lags the most recent nightly so there might be some build errors, please let me know if there are and we'll resolve them.

[okhat]

I'm able to build and use it without issues! Many thanks @cpuhrsch!

I'll explore efficiency and portability. In particular, I hope this would be much faster than naive use of torch.as_strided and that it'll be easy enough to make sure our users don't necessarily have to compile from source [or at least don't have to think about it].

Happy to open a new issue if needed, but a couple of quick question if that's okay:

• Is there support for using nested tensors as an index? Something along the lines of flat_torch_tensor[nested_tensor] -> nested_tensor?

• Is there support for creating a nested tensor from a flat tensor alongside lengths, as in nestedtensor(flat_tensor, lengths)? If not, is there a standard/efficient way to replicate this behavior? Perhaps creating a mask for nested_tensor_from_tensor_mask is efficient enough?

The most efficient way I can think of here is to use torch.as_strided, create a contiguous copy of the tensor, mask out the padding spaces, then use nested_tensor_from_tensor_mask. But that seems pretty expensive!

[cpuhrsch]

@okhat,

Happy to hear you got it to work!

• Is there support for using nested tensors as an index? Something along the lines of flat_torch_tensor[nested_tensor] -> nested_tensor?

Could you give an example of the expected behavior here?

• Is there support for creating a nested tensor from a flat tensor alongside lengths, as in nestedtensor(flat_tensor, lengths)? If not, is there a standard/efficient way to replicate this behavior? Perhaps creating a mask for nested_tensor_from_tensor_mask is efficient enough?

No there is not, but there really should be. In particular this constructor could be nestedtensor.as_strided(input, nested_size) for now. The user provides a Tensor which will be used as the data and a nested_size (a list of torch.Sizes) to pass the shape. What do you think?