Docs: behavior of regular expression pattern caching is unclear

[sethdandridge]

Documentation

The current documentation for the re.compile function mentions that compiled regular expressions are cached, but the wording could be more precise to clarify that this caching also applies to patterns passed as strings to other module functions (e.g., re.search, re.match). This change would help guide developers away from unnecessarily calling re.compile.

This abbreviated code snippet demonstrates the behavior I am referring to:

import re

re.match(r"fo+", "fooooooo")

for cache_entry in re._cache.items():
    print(cache_entry)

#> ((<class 'str'>, 'fo+', 0), re.compile('fo+'))

Linked PRs

• gh-139316: Docs: Clarify behavior of regular expression pattern caching as it relates to re.compile #139317

[AnuragDixitRaipur]

Thanks.

[picnixz]

I don't understand. The docs say:

The compiled versions of the most recent patterns passed to :func:re.compile and the module-level matching functions are cached.

It already mentions the module-level matching functions. What more should it be done? also we say "compiled versions of the most recent patterns" so it should be clear that we're talking about the string patterns here.

[deforestg]

The doc string in question was ambiguous and awkwardly worded. It was linked to me recently and I was confused by it. This is a marked improvement.

[picnixz]

I do not agree here. In what way can it be confusing? what is ambiguous here? unless there is a clear evidence that the wording is ambiguous (e.g., by showing how it could be understood differently), I am closing this as wont fix.

[sethdandridge]

I still maintain that the re docs would benefit from this improvement. The fact that passing a string pattern to any of the matching patterns in re causes the pattern to be internally compiled and cache is surprising behavior. It is therefore, in my opinion, worthy of more explicit explanation.

Furthermore, the note's use of the term "matching function" isn't well-defined. In the module documentation, the term "matching function" is only used once, in the note in question. It would be additionally helpful to users if we clarified that "matching functions" encapsulates most of the typically used functions in re.

[picnixz]

The fact that passing a string pattern to any of the matching patterns in re causes the pattern to be internally compiled and cache is surprising behavior

But this is explained. It is exactly what the sentence says:

The compiled versions of the most recent patterns passed to :func:`re.compile` and the module-level matching functions are cached.

Which part of the sentence is confusing? it says "the compiled versions are cached". In order to compile a pattern, either you pass an already compiled pattern object, and you get the same pattern, or you pass a string/bytes (in which case we cache the compiled versions thereof) and get a (possibly already) cached pattern as well. Since we don't want to distinguish between all possible patterns (compiled, strings, bytes), saying "patterns" is sufficient.

The fact that passing a string pattern to any of the matching patterns in re causes the pattern to be internally compiled and cache is surprising behavior

Ut's explained in the regular docs, but it's also re-explained in the "more gentle" introduction (https://docs.python.org/3/howto/regex.html#module-level-functions) and the wording isn't that different.

Furthermore, the note's use of the term "matching function" isn't well-defined

I think we can decently assume that which functions they are (we do so in other parts of the docs). Exhaustively listing them also adds maintenance burden if we were to add one or deprecate one, and it would make the note way too large (the docs don't like to exhaustively list functions when possible). We also mention "module-level" functions without exactly listing them, and this doesn't cause any issue I think?

---

Also, did this "unclarity" of the docs cause a real issue in practice? if there is one, I would be happy to change them to prevent this, but I don't see what kind of issue it could cause to the reader.