fix: don't import datasets on the inference path

[emiliano-go]

import needle pulled in HuggingFace datasets (and its pyarrow/pandas/fsspec tree) even though nothing on the inference path uses it. The tokenizer symbols that run.py and __init__.py need are defined in needle/dataset/tokenizer.py but were being imported through needle/dataset/dataset.py, which imports datasets at module level.

This repoints those imports at the module where the symbols are actually defined. Same objects, no API or dependency changes.

Closes #40

Changes

File	Change
needle/init.py:14	dataset.dataset → dataset.tokenizer
needle/model/run.py:11	dataset.dataset → dataset.tokenizer
needle/dataset/init.py:2	removed from .dataset import *

The third change is required, not cleanup. needle/dataset/__init__.py executes on any needle.dataset.* import, so it triggers datasets before tokenizer.py is ever reached. Without it, the first two changes have no effect.

Verification

python -c "import sys; import needle; print('datasets' in sys.modules)"
# before: True
# after:  False

• needle.get_tokenizer, needle.generate, needle.load_checkpoint all still resolve
• get_tokenizer is the same object as before (gt1 is gt2)
• from needle.dataset.dataset import ... still works; datasets loads lazily when training code actually runs

Safety

No internal consumer relies on the star re-export. All existing imports are either submodule imports (from ..dataset import dataset as data_mod, from ..dataset import generate as gd) or submodule-qualified (from .dataset import ...), none of which are affected by removing it.

[emiliano-go]

Removing from .dataset import * from needle/dataset/__init__.py means external users

from needle.dataset import SomeTrainingClass

will now get an AttributeError. Internal imports are all safe (submodule-qualified), but for any public downstream consumers, this is a breaking change. Happy to adjust based on your versioning policy, just wanted to flag it explicitly.

Also, to make sure this heavy datasets import never slips back in, I'd recommend adding this quick check to the CI workflow:

python -c "import sys; import needle; assert 'datasets' not in sys.modules, 'datasets leaked into inference path'"

It's cheap (~100ms) and ensures the inference path stays lean indefinitely. I can add it in a separate commit or a follow-up PR, just lmk.