Fix LoggerWrapper inefficiency#1213
Merged
rmjarvis merged 4 commits intoreleases/2.4from Mar 22, 2023
Merged
Conversation
beckermr
approved these changes
Mar 22, 2023
Contributor
beckermr
left a comment
There was a problem hiding this comment.
I don't think there's a good test for this so LGTM!
rmjarvis
commented
Mar 22, 2023
rmjarvis
commented
Mar 22, 2023
Contributor
|
For posterity, here is the timing of a simple test on perlmutter: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We have a class in GalSim config called LoggerWrapper, which is intended to avoid communicating with the main logger via the proxy when we aren't actually going to output anything. E.g. logger.debug statements when the logging level is only INFO or WARNING. This is mostly a big improvement.
However, @beckermr and @jchiang87 have each recently found huge inefficiencies when running on Perlmutter with ~128 processes. I tracked it down to when we make a new LoggerWrapper instance and call
logger.getEffectiveLevel(). This had always been going back to the original proxy logger instance even when we already have a wrapped one. And for lots of processes, this can end up clogging the inter-process communication completely unnecessarily.This PR makes the fairly trivial change that when we're re-wrapping something that is already wrapped, it doesn't call
getEffectiveLevel. It just gets the already received value stored as an attribute. This seems to resolve the blockage that Matt (and I presume Jim) was seeing.