python: replace third-party lazy imports with the standard library#24402
python: replace third-party lazy imports with the standard library#24402tamird wants to merge 3 commits into
Conversation
|
@wing328 could you please take a look? |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
Module getattr is already the cache-miss condition: Python calls it only when the requested name is absent from the module namespace (https://peps.python.org/pep-0562/). During resolution, import_module can insert a same-named child module into that namespace (https://docs.python.org/3/reference/import.html#submodules). Checking the namespace again or using globals().setdefault(name, value) would then retain the child module instead of the exported class. globals()[name] = value must therefore overwrite that intermediate entry after the import succeeds. Later accesses find the cached class directly and never invoke getattr or write again; a separate helper cache would not prevent module-namespace lookup from bypassing it. I benchmarked the actual 842-export Kubernetes client in 36 alternating fresh Python 3.12 processes, keeping the model graph and installed dependencies constant and substituting only the three generated initializer implementations in memory. Median cold import improved from 113.6 to 106.7 ms, and 100,000 cached model accesses from 6.00 to 4.13 ms. First CoreV1Api access was effectively unchanged. Peak RSS increased by approximately 2.1 MB. This measures source-equivalent initializers, not a canonically regenerated Kubernetes release, and does not establish a memory or copy-on-write improvement. — tamirdex |
|
thanks for the PR moving forward, please cc @cbornet (2017/09) @tomplus (2018/10) @krjakbrjak (2023/02) @fa0311 (2023/10) |
|
Thanks, noted. I’ll include the Python-generator maintainers on future generator changes. [ tamirdex ] |
7a07b20 to
3926734
Compare
|
This is rebased and ready to land! |
Python's lazy-import mode depends on lazy-imports and replaces entries in sys.modules, introducing the supply-chain concern tracked in #21831.
Use standard-library getattr and importlib to load API and model modules only when accessed. Retain the explicit typing guards introduced by d8b78d4, cache each export in its real module namespace, and remove the third-party runtime dependency. https://peps.python.org/pep-0562/