-
Notifications
You must be signed in to change notification settings - Fork 39.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PodDisruptionBudget should use ControllerRef #45003
Conversation
@enisoc @caesarxuchao PTAL |
controllerScale := map[types.UID]int32{} | ||
for _, rs := range rss { | ||
// GetDeploymentsForReplicaSet returns an error only if no matching | ||
if controllerRef := controller.GetControllerOf(pod); controllerRef != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can remove the TODO in line 164 now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That TODO refers to an improvement made possible by ControllerRef, but we are not attempting to implement that improvement here, so the TODO still stands.
@caesarxuchao will multiple owner refs in a Pod break cascading deletion for controllers? Now I have to delete both the StatefulSet and PDB in order to delete the Pods? |
@Kargakis, I didn't look at this PR. If there are two ownerRefs in the Pod referring to the PDB and StatufulSet respectively, then Pod will be garbage collected after both PDB and StatuefulSet are deleted. |
controllerScale[rs.UID] = *(rs.Spec.Replicas) | ||
} | ||
} | ||
for uid, scale := range controllerScale { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is only one value in controllerScale, why use a loop?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@CaoShuFeng yes will fix once i fix the tests
if err != nil { | ||
return cas, nil | ||
} | ||
// GetPodStatefulSets returns an error only if no StatefulSets are found. We |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment needs update.
controllerScale := map[types.UID]int32{} | ||
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil { | ||
deployment, err := dc.dLister.Deployments(rs.Namespace).Get(controllerRef.Name) | ||
// GetDeploymentsForReplicaSet returns an error only if no matching |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment needs update.
@Kargakis @caesarxuchao This is not going to create an OwnerRef from Pod to PDB. This is only about PDB looking at existing ControllerRefs instead of inferring ownership of Pods through selectors. |
controllerScale := map[types.UID]int32{} | ||
for _, rs := range rss { | ||
// GetDeploymentsForReplicaSet returns an error only if no matching | ||
if controllerRef := controller.GetControllerOf(pod); controllerRef != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That TODO refers to an improvement made possible by ControllerRef, but we are not attempting to implement that improvement here, so the TODO still stands.
if controllerRef := controller.GetControllerOf(pod); controllerRef != nil { | ||
rs, err := dc.rsLister.ReplicaSets(pod.Namespace).Get(controllerRef.Name) | ||
if err != nil { | ||
return cas, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please retain the comment explaining why we are ignoring the error (returning nil
).
Specifically: the only possible error when fetching from the local cache is NotFound. If the referenced ReplicaSet does not exist, we consider that as successfully finding that the Pod is not controlled by any ReplicaSets.
@@ -165,6 +165,37 @@ func newPodDisruptionBudget(t *testing.T, minAvailable intstr.IntOrString) (*pol | |||
return pdb, pdbName | |||
} | |||
|
|||
func newPodOwnedByRc(t *testing.T, name string, rc *v1.ReplicationController) (*v1.Pod, string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Go style, we capitalize acronyms: newPodOwnedByRC
. Same for RS
, SS
, etc.
_, err := dc.dLister.GetDeploymentsForReplicaSet(rs) | ||
if err == nil { // A deployment was found, so this finder will not count this RS. | ||
continue | ||
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this may be why the test is failing, at least for RS. If controllerRef == nil
, we actually want to return the RS without any further checking.
The idea here is, if the RS is a top-level controller, we return it. If the RS is controlled by a Deployment, we don't return it, because we instead want the Deployment itself to be returned by getPodDeployments
. If the RS is controlled by something that isn't a Deployment, we should still return it because we don't know the semantics of that unknown thing.
There's an opportunity now to simplify and optimize this whole process. We can actually get rid of the finders and have one function called something like getPodControllerScale
. It would check the Pod's ControllerRef and figure out the scale somehow depending on the type of controller. Let me know if you want to pursue that longer-term fix and we can talk about it. It's also fine if you just want to finish this intermediate fix first and leave the refactoring for later.
for _, d := range ds { | ||
controllerScale[d.UID] = *(d.Spec.Replicas) | ||
controllerScale := map[types.UID]int32{} | ||
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Go style, we try to keep the "error-free" path as unindented as possible. In this case, that means:
controllerRef := ...
if controllerRef == nil {
return cas, nil
}
deployment, err := ...
deployment, err := dc.dLister.Deployments(rs.Namespace).Get(controllerRef.Name) | ||
// GetDeploymentsForReplicaSet returns an error only if no matching | ||
// deployments are found. In that case we skip this ReplicaSet. | ||
if err == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similarly here:
if err != nil {
return cas, nil
}
``
@enisoc thanks i think i figured the tests as well. I will send out an updated PR in a couple days. |
92e1f5d
to
26196f3
Compare
@@ -173,94 +174,89 @@ func (dc *DisruptionController) finders() []podControllerFinder { | |||
dc.getPodStatefulSets} | |||
} | |||
|
|||
var ( | |||
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet") | |||
controllerKindSS = v1beta1.SchemeGroupVersion.WithKind("StatefulSet") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
statefulset is part of the apps group.
var ( | ||
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet") | ||
controllerKindSS = v1beta1.SchemeGroupVersion.WithKind("StatefulSet") | ||
controllerKindRC = v1beta1.SchemeGroupVersion.WithKind("ReplicationController") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rc is part of the core legacy group
_, err := dc.dLister.GetDeploymentsForReplicaSet(rs) | ||
if err == nil { // A deployment was found, so this finder will not count this RS. | ||
continue | ||
casSlice := []controllerAndScale{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var casSlice []controllerAndScale
return nil, nil | ||
} | ||
controllerRef := controller.GetControllerOf(rs) | ||
if controllerRef == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if controllerRef == nil || controllerRef.Kind != controllerKindDep.Kind {
for _, s := range ss { | ||
controllerScale[s.UID] = *(s.Spec.Replicas) | ||
} | ||
casSlice := []controllerAndScale{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please change this for all instances.
765261f
to
95ac017
Compare
@k8s-bot verify test this |
if err != nil { | ||
return nil, nil | ||
} | ||
controllerRef := controller.GetControllerOf(rs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After fetching the RS by name, check that the UID matches controllerRef.UID
. If it doesn't match, the RS you fetched doesn't actually own the pod; it just has the same name as a previously-deleted RS that owned the pod. In that case, you can return nil, nil
since the pod has no RS owner.
if controllerRef.Kind == controllerKindRS.Kind { | ||
rs, err := dc.rsLister.ReplicaSets(pod.Namespace).Get(controllerRef.Name) | ||
if err != nil { | ||
return nil, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add back the comment that explains why we ignore err
: It's because the only possible error returned by the cache is NotFound, and that means there is no RS for this pod.
} | ||
controllerRef := controller.GetControllerOf(rs) | ||
if controllerRef == nil || controllerRef.Kind != controllerKindDep.Kind { | ||
casSlice = append(casSlice, controllerAndScale{rs.UID, *(rs.Spec.Replicas)}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Go style suggestion: This line is the end of the "happy path" (everything went as expected - we found an RS). Ideally it should be as unindented as possible, and unexpected things should result in early return
statements. This way the "happy path" flows straight down rather than becoming more deeply nested at each step.
As an example, this function could be structured like:
...
if controllerRef == nil {
return nil, nil
}
if controllerRef.Kind != controllerKindRS.Kind {
return nil, nil
}
rs, err := ...
if err != nil {
// The only possible error is NotFound, which is ok here.
return nil, nil
}
if rs.UID != controllerRef.UID {
return nil, nil
}
...
if controllerRef != nil && controllerRef.Kind == controllerKindDep.Kind {
// Skip RS if it's controlled by a Deployment.
return nil, nil
}
return []controllerAndScale{...}
Notice there's no nesting and everything flows straight down.
ss, err := dc.ssLister.StatefulSets(pod.Namespace).Get(controllerRef.Name) | ||
if err != nil { | ||
return nil, nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check UID after Get.
if controllerRef.Kind == controllerKindSS.Kind { | ||
ss, err := dc.ssLister.StatefulSets(pod.Namespace).Get(controllerRef.Name) | ||
if err != nil { | ||
return nil, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add back comment about NotFound error being ok.
rc, err := dc.rcLister.ReplicationControllers(pod.Namespace).Get(controllerRef.Name) | ||
if err != nil { | ||
return nil, nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check UID after Get.
add(t, dc.rcStore, rc) | ||
dc.sync(pdbName) | ||
|
||
fmt.Println("reeached here 1") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove raw print or convert to t.Log()
.
dc.sync(pdbName) | ||
|
||
// No controllers yet => no disruption allowed | ||
ps.VerifyDisruptionAllowed(t, pdbName, 0) | ||
|
||
rc, _ := newReplicationController(t, 1) | ||
rc.Name = "rc 1" | ||
for i := 0; i < podCount; i++ { | ||
updatePodOwnerToRc(t, pods[i], rc) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this test be split into multiple test cases, so we don't modify pods that were already added to the pod store? In my experience, that can lead to flaky/racy tests because you are changing things you already gave to the pod store, which it is not designed to support.
If these need to be in one test case/func, is it possible to add new pods rather than changing existing ones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
took care of all other comments. will try this and see how it works. Not very familiar with this framework
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR LGTM other than the potential flakiness here. If you want, we can just try to measure whether this is going to be flaky and submit it as is if it passes.
One way to do that is something like go test -count 1000
or go test -race -count 1000
. You might want to start with a smaller count and work up to what your machine can handle. It seems that Go sometimes runs the tests concurrently and I've locked up my workstation before by putting a count that was too high. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
go test ./pkg/controller/disruption/... --count 300 --race
race: limit on 8192 simultaneously alive goroutines is exceeded, dying
FAIL k8s.io/kubernetes/pkg/controller/disruption 16.577s
go test ./pkg/controller/disruption/... --count 250 --race
ok k8s.io/kubernetes/pkg/controller/disruption 16.575s
I cant go beyond 250 on my mac. Is there a way to request this count with the prbot ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try running higher counts, but without -race
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
go test ./pkg/controller/disruption/... --count 300
ok k8s.io/kubernetes/pkg/controller/disruption 2.348s
go test ./pkg/controller/disruption/... --count 400
ok k8s.io/kubernetes/pkg/controller/disruption 3.047s
go test ./pkg/controller/disruption/... --count 600
ok k8s.io/kubernetes/pkg/controller/disruption 4.573s
go test ./pkg/controller/disruption/... --count 900
ok k8s.io/kubernetes/pkg/controller/disruption 6.864s
go test ./pkg/controller/disruption/... --count 1200
ok k8s.io/kubernetes/pkg/controller/disruption 9.862s
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Let me know when you've resolved the merge conflicts and I will LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@enisoc resolved conflicts.
add(t, dc.rcStore, rc) | ||
dc.sync(pdbName) | ||
|
||
fmt.Println("reeached here 1, %v", len(pods[0].OwnerReferences)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove or convert to t.Log()
.
} | ||
|
||
func updatePodOwnerToRs(t *testing.T, pod *v1.Pod, rs *extensions.ReplicaSet) { | ||
var controllerKind = apps.SchemeGroupVersion.WithKind("ReplicaSet") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you use the vars you already defined in the other file?
65c77d6
to
e7ac592
Compare
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet") | ||
controllerKindSS = apps.SchemeGroupVersion.WithKind("StatefulSet") | ||
controllerKindRC = v1.SchemeGroupVersion.WithKind("ReplicationController") | ||
controllerKindDep = v1beta1.SchemeGroupVersion.WithKind("Deployment") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also have Deployments in the apps group. I think you need to use both but I am not fairly certain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, you just compare UID and Kind so it may not be needed after all
} | ||
|
||
return cas, nil | ||
casSlice = append(casSlice, controllerAndScale{ss.UID, *(ss.Spec.Replicas)}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it's only one parent that you are returning now, switch all these helpers to return a single object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Kargakis i was hoping to do that change in a subsequent PR, since that would also involve modifying the callers and understanding that logic. Is that ok ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure
@k8s-bot pull-kubernetes-e2e-gce-etcd3 test this |
/lgtm |
/assign @derekwaynecarr |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: derekwaynecarr, enisoc, krmayankk
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
@k8s-bot pull-kubernetes-federation-e2e-gce test this |
Automatic merge from submit-queue (batch tested with PRs 45518, 46127, 46146, 45932, 45003) |
Automatic merge from submit-queue simplify disruption controller finder logic **What this PR does / why we need it**: Address some comments from #45003 and simplify the PDB controller logic as part of issue #42284 @enisoc @Kargakis @caesarxuchao Also it feels like we can get rid of the finders all together since with controller ref, each pod has only controller. Let me know if i should remove that finders all together ?
Fixes #42284