Skip to content

Commit 27239f2

Browse files
Update untokenize.rst documentation to reflect current implementation
1 parent 1daad8a commit 27239f2

1 file changed

Lines changed: 12 additions & 6 deletions

File tree

Doc/library/tokenize.rst

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -85,17 +85,23 @@ write back the modified script.
8585
.. function:: untokenize(iterable)
8686

8787
Converts tokens back into Python source code. The *iterable* must return
88-
sequences with at least two elements, the token type and the token string.
89-
Any additional sequence elements are ignored.
88+
sequences with either two or five elements.
9089

9190
The result is guaranteed to tokenize back to match the input so that the
92-
conversion is lossless and round-trips are assured. The guarantee applies
93-
only to the token type and token string as the spacing between tokens
94-
(column positions) may change.
91+
conversion is lossless and round-trips are assured.
92+
93+
If *iterable* returns sequences with two elements (the token type and token
94+
string), the result will tokenize back to the same token types and strings as
95+
the input, but the spacing between tokens (column positions) may change.
96+
97+
If *iterable* returns sequences with five elements
98+
(``type token string start end line``), the column positions are preserved
99+
and the result will tokenize back to match the input exactly.
100+
95101

96102
It returns bytes, encoded using the :data:`~token.ENCODING` token, which
97103
is the first token sequence output by :func:`.tokenize`. If there is no
98-
encoding token in the input, it returns a str instead.
104+
encoding token in the input, it returns a :class:`str` instead.
99105

100106

101107
:func:`.tokenize` needs to detect the encoding of source files it tokenizes. The

0 commit comments

Comments
 (0)