@@ -85,17 +85,23 @@ write back the modified script.
8585.. function :: untokenize(iterable)
8686
8787 Converts tokens back into Python source code. The *iterable * must return
88- sequences with at least two elements, the token type and the token string.
89- Any additional sequence elements are ignored.
88+ sequences with either two or five elements.
9089
9190 The result is guaranteed to tokenize back to match the input so that the
92- conversion is lossless and round-trips are assured. The guarantee applies
93- only to the token type and token string as the spacing between tokens
94- (column positions) may change.
91+ conversion is lossless and round-trips are assured.
92+
93+ If *iterable * returns sequences with two elements (the token type and token
94+ string), the result will tokenize back to the same token types and strings as
95+ the input, but the spacing between tokens (column positions) may change.
96+
97+ If *iterable * returns sequences with five elements
98+ (``type token string start end line ``), the column positions are preserved
99+ and the result will tokenize back to match the input exactly.
100+
95101
96102 It returns bytes, encoded using the :data: `~token.ENCODING ` token, which
97103 is the first token sequence output by :func: `.tokenize `. If there is no
98- encoding token in the input, it returns a str instead.
104+ encoding token in the input, it returns a :class: ` str ` instead.
99105
100106
101107:func: `.tokenize ` needs to detect the encoding of source files it tokenizes. The
0 commit comments