-
Notifications
You must be signed in to change notification settings - Fork 39.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Total priority overflow check #45122
Total priority overflow check #45122
Conversation
Hi @ravisantoshgudimetla. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
/cc @jayunit100 |
@@ -39,3 +39,22 @@ func TestAlgorithmNameValidation(t *testing.T) { | |||
} | |||
} | |||
} | |||
|
|||
func TestPriorityOverFlow(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also Add a unit test to confirm it doesnt overflow for valid conditions ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought about that but I guess that should be part of other test that calls the checkOverFlow function. I don't have strong opinion on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
definetly should have a test confirming it doesnt always overflow. that would be pretty bad :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ack..
return configs, nil | ||
} | ||
|
||
// checkOverFlow checks if totalPriority can overflow after adding it with weight. | ||
func checkOverFlow(totalPriority int, weight int) bool { | ||
if totalPriority == MaxInt || weight == MaxInt { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- is this redundant to the weight check ? After all if
totalPriority
overflows, thentotalPriority+ weight
overflows as well in all cases, right ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once it overflows, no way it can exceed, so == should be fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but why check == if we're planning on checking below?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, by the time the function is called, one of the variables could have already at MaxInt and after that there is no point in doing an addition as it would throw a buffer overflow, so its kind of
- Failing early.
- The computation wouldn't cause an error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Youre right regarding >=
, ill delete that to not confuse people :) .
final thought is that IMO if this whole function can be replaced with return (total+weight)-weight != 0
, it shouldn't be a function at all, b/c one line of code (excluding old school perl code golf examples) is usually better then 5.
will let you guys decide though...
var totalPriority int | ||
for _, config := range configs { | ||
// Checks for overflow and returns errors. | ||
// TODO: Need to check overflow for weight*10. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
weight*10 is being checked below, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right but we need to have check for * but that's for the calling function.(Means if we have a unit test for getPriorityFunctionConfigs.)
|
||
func TestPriorityOverFlow(t *testing.T) { | ||
var a, b int | ||
a = 5000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
call a
total, and call b
weight...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. The only thought that I had was, this could be used anywhere, so did not name variables to reflect weight and total.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually just delete a/b we dont need them do we
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok..
return true | ||
} | ||
sum := totalPriority + weight | ||
return sum-totalPriority != weight |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so this is just return totalPriority+weight-totalPriority !=weight
i think thats pretty much always true... ? in this universe ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right but for readability..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gotcha, so its overflow i see whats going on here now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just make it oneline, and add a comment about how in go we have to test for int overflow additions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gotcha makes sense now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'd prefer
return (totalPri + weight)-totalPriority != weight
``` since that makes it obvious that your testing the absurd boundary overflow. but i guess just an opinion.
return configs, nil | ||
} | ||
|
||
// checkOverFlow checks if totalPriority can overflow after adding it with weight. | ||
func checkOverFlow(totalPriority int, weight int) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO we should call this function checkIntegerOverflow
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, makes sense
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
checkPriorityOverflow
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess, if we want to use this in someother location, its better to generalize the name to checkIntegerOverflow
as this function can check for any 2 int's overflow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if so, I'd like to move it to util
and export it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or if the check is simple enough, this func is not necessary. But we need to add unit test for priority config check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The check may be simple enough, my only concern is I have to write unit tests for getPriorityFunctionConfigs, which is kind of orthogonal to issue. IMO, the best thing to do is leave this function as is here now and move it to utils once we find other functions that can use this. I am open to other suggestions too.
I think this mostly makes sense, minor comments b/c my head spins when thinking about overflow. Also I don't know the background of the priority normalization goals, so best if someone else finishes the review... @kubernetes/sig-scheduling-pr-reviews . |
@k8s-bot test this please |
if totalPriority == MaxInt || weight == MaxInt { | ||
return true | ||
} | ||
sum := totalPriority + weight |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Go, I'm not sure whether sum
is still a valid value after overflow; in C, the sum
maybe randoms after overflow (e.g. checking by interrupt or others) in some arch.
So I'd prefer to
return weight >= MAX_INT || totalPriority >= MAX_INT || weight >= MAX_INT - totalPriority
or just weight > MAX_INT - totalPriority
(when weight
will > MAX_INT
?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(when weight will > MAX_INT?)
Thats pretty hypothetical case but this logic in general seems concise. I will use this. Thanks @k82cn. Just to be clear, since weight is always +ve.
for _, config := range configs { | ||
// Checks for overflow and returns errors. | ||
// TODO: Need to check overflow for weight*10. | ||
if checkOverFlow(totalPriority, config.Weight*10) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like make 10
as MaxPriority
.
My understand is to make priority comparable, e.g. we can not compare |
e73dea4
to
d6c6ff8
Compare
@k82cn @jayunit100 , PTAL. I have addressed most of comments except for location of this function. LMK, if you have any other comments. |
d6c6ff8
to
1d39501
Compare
@@ -87,6 +87,9 @@ var ( | |||
|
|||
const ( | |||
DefaultProvider = "DefaultProvider" | |||
MaxUint = ^uint(0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we externalized these or add MaxUint
and MaxInt
to math.MaxInt
math.MaxUInt
? I'm pretty sure we can, and If not, we should add them to a util somewhere.
var totalPriority int | ||
for _, config := range configs { | ||
// Checks for overflow and returns errors. | ||
// TODO: Need to check overflow for weight*MaxPriority. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets accomplish this TODO in this PR, ... just add a third arg to your overflow function below, so that it checks everything for us.....
if totalPriority, err := addPrioritySafely(totalPriority, config.Weight, MaxPriority); err != nil {
return nil, fmt.Errorf("Total priority of priority functions has overflown %v", err)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree that we should handle overflow for weight*MaxPriority
instead of TODO
.
And I'd suggest to move this check to plugin/pkg/scheduler/api/validation/validation.go
; as it also check weight against negative number.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
haha this simple PR is getting pretty interesting :). Yeah, k82cn's idea is reasonable, if we can move this thing into a generic place (make sure to create corresponding unit test) thats great.
return configs, nil | ||
} | ||
|
||
// checkIntegerOverFlow checks if totalPriority can overflow after adding it with weight. | ||
func checkIntegerOverFlow(totalPriority int, weight int) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, sorry, now that I better understand whats going on here, im going to suggest you call this addPrioritySafely
,
- Check weight*Maxpriority multiplies w/o overflow.
- then check that totalPriority increments w/o overflow.
- Return error if fails.
- Otherwise return the new added on value of totalPriority
That will be super easy to understand. Sorry about changing my mind :).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jayunit100 , @k82cn, I am fine with making those changes but AFAIU, it seems weights can never overflow, even if it overflows, shouldn't we check them in functions where they were getting updated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was just suggesting that the entire addPriority...
operation should handle the TODO as well as the other stuff all in one place, and return a new value representing the result of the operation. That makes it really easy to read, unit test, and so on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with you but the overflow can happen at other places as well where weight is getting updated. So, I was suggesting that we should have checks there instead of here, as this is the last step in calculation. (I meant, we need to fail earlier than this.).
1d39501
to
a7c9d99
Compare
@k82cn, @jayunit100 PTAL. I moved the check to validation.go. Now, it checks for weight*10 overflow as well. The function could have been much more concise but it may effect readability. LMK, your thoughts on it. |
sorry for front and rear, here's the diff in my mind (just code style, no functional comments):
here's a draft diff for your reference: https://gist.github.com/k82cn/ee104da32ae615c0458fdcecc693e918 |
@k82cn : Thanks for the feedback. Please see my responses inline.
Reasonable. I can make the change.
The problem here is if you compute MaxInt/10 and multiply it with 10, we will not get MaxInt back.
Sure..
TBH, I did not think about this approach as I took some piece of code from the closed PR. But, after looking at the code base, it seems that we are doing a lazy evaluation of priority weights. The idea that I had is to fail fast and early and wanted to change the code(in another PRs) to reflect that. But, if you and others insist, I can leave this as is and write the validateSelectedConfigs func.
I wanted to avoid writing test cases for getPriorityFunctionConfigs as that is orthogonal to the current set of changes and there is another issue that you are shepherding to improve test suite coverage. Writing test case for validateSelectedConfigs, if we choose the above method seems reasonable. |
If so, then
What's the gap? it just extra a bit of code into a func.
I think we did not cover the major logic in |
ill defer to k82cn for the rest of the changes, this is a simple PR so don't need too many cooks :). |
@k8s-bot ok to test |
ping @ravisantoshgudimetla :). |
@k82cn, Apologies for delay. I will update this PR in next 2-3 days. |
@ravisantoshgudimetla , thanks very much :). |
a7c9d99
to
7ae3136
Compare
@k82cn, I have addressed most of your comments. LMK, if you want any other changes. |
@@ -29,8 +29,8 @@ func ValidatePolicy(policy schedulerapi.Policy) error { | |||
var validationErrors []error | |||
|
|||
for _, priority := range policy.Priorities { | |||
if priority.Weight <= 0 { | |||
validationErrors = append(validationErrors, fmt.Errorf("Priority %s should have a positive weight applied to it", priority.Name)) | |||
if priority.Weight <= 0 || priority.Weight >= schedulerapi.MaxWeight { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prefer to priority.Weight > schedulerapi.MaxWeight
, it'll align with checking against MaxTotalPriority
.
/lgtm That's great; minor comments. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: k82cn, ravisantoshgudimetla Assign the PR to them by writing
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
Automatic merge from submit-queue (batch tested with PRs 44727, 45409, 44968, 45122, 45493) |
What this PR does / why we need it:
Which issue this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close that issue when PR gets merged): fixes #24720Special notes for your reviewer:
@adohe. I have borrowed some parts of your code in the closed PR and created this one.
Release note: