gh-121084: Fix test_typing random leaks #121360
Conversation
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses.
|
Without this change, test_typing leaks randomly. Examples of output:
With this change, it no longer leaks. |
There was a problem hiding this comment.
Do we have any idea why typing is particularly affected here? It's it because of some unusual things we're doing with ABCs in the typing module, or is it a general issue with ABC caches? If the latter, maybe we should be doing this for every module, not just typing?
test_typing creates many classes, register them in ABC, check for subclasses, etc. It uses heavily the ABC caches. |
|
So do we think that this is something that could happen to any module that has a lot of ABCs in it? If we added a few more ABCs to |
There was a problem hiding this comment.
I'm fine with this if it fixes the tests, but I'd prefer it if we got to the bottom of what the root cause is here. If it's something that could affect any module with lots of ABCs, then that suggests a more general fix might be warranted. If it's something that specifically affects typing because of something we're doing in that module, then we could probably do a more targeted fix.
Refleak check is both strict and stupid. It counts beans. If a function allocates one memory block and doesn't release it, refleak check reports a memory leak. It doesn't understand if the memory block belongs to a cache or if it's "a real leak". That's why we need some workarounds like that which are an exhaustive list of caches that should be cleared after after run. I don't think that anything is wrong with typing or test_typing. They are populating caches, that's good. Internals rely on the garbage collector and weak references which make test_typing a little bit less deterministic when it comes to "refleak check". Ah, there is one thing that I noticed. It seems like the ABC cache is a set of weak references, but dead references are never removed from the set. That's one thing that could be enhanced and make this problem less important. |
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses. (cherry picked from commit 5f660e8) Co-authored-by: Victor Stinner <vstinner@python.org>
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses. (cherry picked from commit 5f660e8) Co-authored-by: Victor Stinner <vstinner@python.org>
|
GH-121372 is a backport of this pull request to the 3.12 branch. |
|
GH-121373 is a backport of this pull request to the 3.13 branch. |
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses.
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses.
Clear typing ABC caches when running tests for refleaks (-R option): call _abc_caches_clear() on typing abstract classes and their subclasses.