Skip to content

fix: Fix snapshots handling#692

Merged
Pijukatel merged 9 commits intomasterfrom
memory-usage-fix
Nov 15, 2024
Merged

fix: Fix snapshots handling#692
Pijukatel merged 9 commits intomasterfrom
memory-usage-fix

Conversation

@Pijukatel
Copy link
Copy Markdown
Collaborator

@Pijukatel Pijukatel commented Nov 13, 2024

Small fixes to snapshot handling:

  • Ensure time ordering of snapshots.
  • Remove fake time weight for samples from same time.
  • Remove unnecessary reversing of snapshot.
  • Add test for memory and cpu snapshot ordering.

Aditional reverse was changing order of snapshots and the rest of code expected latest snapshots last.
Simplified ratio calculation.
Remove fake 0.001s for zero timed measurements. Snapshot of zero duration should not contribute to ratio calculation.
exploring order of samples and the lack of guarantee of their ordering
Original with extra debug
-Ensure time ordering.
-Remove fake time weight for samples from same time.
-Remove unnecesary reversing of snapshot.
Add test for memory and cpu snapshot ordering.
@Pijukatel Pijukatel added the bug Something isn't working. label Nov 13, 2024
@github-actions github-actions bot added this to the 102nd sprint - Tooling team milestone Nov 13, 2024
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Nov 13, 2024
Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

bisect.insort does not support 'key' keyword untill python 3.10
sortedcollections are already in part of project, so no extra dependency
@Pijukatel Pijukatel marked this pull request as draft November 13, 2024 15:49
@Pijukatel Pijukatel marked this pull request as ready for review November 14, 2024 09:19
@vdusek vdusek changed the title fix: Snapshots handling fix: Fix snapshots handling Nov 14, 2024
Copy link
Copy Markdown
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I have just a few details... And, could you please also fill in the Snapshotter class docstring as I wrote here. Now is a great time to capture it while it's still fresh in your mind.

Remove unused compute_weighted_avg and tests.
Copy link
Copy Markdown
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last type-inference thing, otherwise LGTM. Let's wait for Honza's approval here as well though.

Copy link
Copy Markdown
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just a couple suggestions mostly regarding tests.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a huge fan of accessing the private interface of Snapshotter to mock the snapshots. There must be a better way 🙂 BTW, I know the original code also touched private stuff, but this PR is all about small improvements...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO out of scope of this issue. Figuring out a better way of testing the Snapshotter should be part of #73.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file also has a lot of private member accesses.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing tests were already doing it. Newly added test does not access a single private member. I would suggest to limit scope of this and refactor the older tests in different change.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We have a dedicated issue to it #73.

raise ValueError('Failed to compute weighted average for the sample.') from exc

return LoadRatioInfo(limit_ratio=threshold, actual_ratio=round(weighted_avg, 3))
time = (current.created_at - previous.created_at).total_seconds()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make sure that time is positive here? (non-negative? not sure)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explicit exception now.

Comment on lines +184 to +185
overloaded_time = 0.0
non_overloaded_time = 0.0
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very minor nit, but it may be clearer to just track overloaded_time and total_time.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did halfway change. I think it is equally readable If I track both overloaded_time and total_time then I have to do some extra sum on each cycle. Super insignificant, but since I already think both approaches are kind of the same, this tips the scales for my decision.

@janbuchar janbuchar self-requested a review November 15, 2024 12:03
Copy link
Copy Markdown
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Pijukatel Pijukatel merged commit 4016c0d into master Nov 15, 2024
@Pijukatel Pijukatel deleted the memory-usage-fix branch November 15, 2024 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants