Remove external untokenize package. Use tokenize from the Python standard library.

[tdenewiler]

No description provided.

[tdenewiler]

This is required for Python 3.14. The untokenize package used by docformatter has not received any new commits in 6 years.

Tested with

docker run -it python:3.14 bash
cd /tmp/
git clone https://github.com/tdenewiler/docformatter.git
cd docformatter
git checkout untokenize-std-lib
pip install tox
tox -e py314

All tests are passing. If you stay on master branch and do not checkout the new branch then those same tests fail.

[tdenewiler]

I ran the tox tests on Python 3.10, 3.11, 3.12, 3.13, and 3.14.

[adamtheturtle]

The key insight here is that untokenize isn't even used. It is an unnecessary dependency.

FWIW I recommend using deptry to find unused dependencies.

[jonathan343]

+1 for removing this unused dependency

[adamtheturtle]

@weibullguy Hopefully this is a very clear "merge", and a release to fix 3.14 would be welcome

[jonathan343]

I'll probably look into the pydocstyle plugin for ruff. It supports a pep257 convention which seems to actually provide more coverage than this repo. Posting incase anyone else is looking for an alternative

[weibullguy]

I'll probably look into the pydocstyle plugin for ruff. It supports a pep257 convention which seems to actually provide more coverage than this repo. Posting incase anyone else is looking for an alternative

Does this plugin format docstrings or just verify them against various standards?

[adamtheturtle]

FWIW even with this PR, on 3.14 docformatter adds extraneous newlines.