Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PodDisruptionBudget should use ControllerRef #45003

Merged
merged 1 commit into from
May 25, 2017

Conversation

krmayankk
Copy link

@krmayankk krmayankk commented Apr 27, 2017

Fixes #42284

PodDisruptionBudget now uses ControllerRef to decide which controller owns a given Pod, so it doesn't get confused by controllers with overlapping selectors.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 27, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Apr 27, 2017
@krmayankk
Copy link
Author

@enisoc @caesarxuchao PTAL

@davidopp davidopp assigned mml and unassigned davidopp and mwielgus Apr 27, 2017
controllerScale := map[types.UID]int32{}
for _, rs := range rss {
// GetDeploymentsForReplicaSet returns an error only if no matching
if controllerRef := controller.GetControllerOf(pod); controllerRef != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove the TODO in line 164 now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That TODO refers to an improvement made possible by ControllerRef, but we are not attempting to implement that improvement here, so the TODO still stands.

@0xmichalis
Copy link
Contributor

@caesarxuchao will multiple owner refs in a Pod break cascading deletion for controllers? Now I have to delete both the StatefulSet and PDB in order to delete the Pods?

@caesarxuchao
Copy link
Member

@Kargakis, I didn't look at this PR. If there are two ownerRefs in the Pod referring to the PDB and StatufulSet respectively, then Pod will be garbage collected after both PDB and StatuefulSet are deleted.

controllerScale[rs.UID] = *(rs.Spec.Replicas)
}
}
for uid, scale := range controllerScale {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only one value in controllerScale, why use a loop?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CaoShuFeng yes will fix once i fix the tests

if err != nil {
return cas, nil
}
// GetPodStatefulSets returns an error only if no StatefulSets are found. We
Copy link
Contributor

@CaoShuFeng CaoShuFeng Apr 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment needs update.

controllerScale := map[types.UID]int32{}
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil {
deployment, err := dc.dLister.Deployments(rs.Namespace).Get(controllerRef.Name)
// GetDeploymentsForReplicaSet returns an error only if no matching
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment needs update.

@enisoc enisoc assigned enisoc and unassigned mml May 12, 2017
@enisoc
Copy link
Member

enisoc commented May 12, 2017

@Kargakis @caesarxuchao This is not going to create an OwnerRef from Pod to PDB. This is only about PDB looking at existing ControllerRefs instead of inferring ownership of Pods through selectors.

controllerScale := map[types.UID]int32{}
for _, rs := range rss {
// GetDeploymentsForReplicaSet returns an error only if no matching
if controllerRef := controller.GetControllerOf(pod); controllerRef != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That TODO refers to an improvement made possible by ControllerRef, but we are not attempting to implement that improvement here, so the TODO still stands.

if controllerRef := controller.GetControllerOf(pod); controllerRef != nil {
rs, err := dc.rsLister.ReplicaSets(pod.Namespace).Get(controllerRef.Name)
if err != nil {
return cas, nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please retain the comment explaining why we are ignoring the error (returning nil).

Specifically: the only possible error when fetching from the local cache is NotFound. If the referenced ReplicaSet does not exist, we consider that as successfully finding that the Pod is not controlled by any ReplicaSets.

@@ -165,6 +165,37 @@ func newPodDisruptionBudget(t *testing.T, minAvailable intstr.IntOrString) (*pol
return pdb, pdbName
}

func newPodOwnedByRc(t *testing.T, name string, rc *v1.ReplicationController) (*v1.Pod, string) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Go style, we capitalize acronyms: newPodOwnedByRC. Same for RS, SS, etc.

_, err := dc.dLister.GetDeploymentsForReplicaSet(rs)
if err == nil { // A deployment was found, so this finder will not count this RS.
continue
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this may be why the test is failing, at least for RS. If controllerRef == nil, we actually want to return the RS without any further checking.

The idea here is, if the RS is a top-level controller, we return it. If the RS is controlled by a Deployment, we don't return it, because we instead want the Deployment itself to be returned by getPodDeployments. If the RS is controlled by something that isn't a Deployment, we should still return it because we don't know the semantics of that unknown thing.

There's an opportunity now to simplify and optimize this whole process. We can actually get rid of the finders and have one function called something like getPodControllerScale. It would check the Pod's ControllerRef and figure out the scale somehow depending on the type of controller. Let me know if you want to pursue that longer-term fix and we can talk about it. It's also fine if you just want to finish this intermediate fix first and leave the refactoring for later.

for _, d := range ds {
controllerScale[d.UID] = *(d.Spec.Replicas)
controllerScale := map[types.UID]int32{}
if controllerRef := controller.GetControllerOf(rs); controllerRef != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Go style, we try to keep the "error-free" path as unindented as possible. In this case, that means:

controllerRef := ...
if controllerRef == nil {
  return cas, nil
}
deployment, err := ...

deployment, err := dc.dLister.Deployments(rs.Namespace).Get(controllerRef.Name)
// GetDeploymentsForReplicaSet returns an error only if no matching
// deployments are found. In that case we skip this ReplicaSet.
if err == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly here:

if err != nil {
  return cas, nil
}
``

@krmayankk
Copy link
Author

@enisoc thanks i think i figured the tests as well. I will send out an updated PR in a couple days.

@krmayankk krmayankk force-pushed the garbage branch 3 times, most recently from 92e1f5d to 26196f3 Compare May 14, 2017 04:38
@@ -173,94 +174,89 @@ func (dc *DisruptionController) finders() []podControllerFinder {
dc.getPodStatefulSets}
}

var (
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet")
controllerKindSS = v1beta1.SchemeGroupVersion.WithKind("StatefulSet")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

statefulset is part of the apps group.

var (
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet")
controllerKindSS = v1beta1.SchemeGroupVersion.WithKind("StatefulSet")
controllerKindRC = v1beta1.SchemeGroupVersion.WithKind("ReplicationController")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rc is part of the core legacy group

_, err := dc.dLister.GetDeploymentsForReplicaSet(rs)
if err == nil { // A deployment was found, so this finder will not count this RS.
continue
casSlice := []controllerAndScale{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var casSlice []controllerAndScale

return nil, nil
}
controllerRef := controller.GetControllerOf(rs)
if controllerRef == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if controllerRef == nil || controllerRef.Kind != controllerKindDep.Kind {

for _, s := range ss {
controllerScale[s.UID] = *(s.Spec.Replicas)
}
casSlice := []controllerAndScale{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var ...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change this for all instances.

@krmayankk krmayankk force-pushed the garbage branch 2 times, most recently from 765261f to 95ac017 Compare May 18, 2017 07:28
@krmayankk
Copy link
Author

@k8s-bot verify test this

if err != nil {
return nil, nil
}
controllerRef := controller.GetControllerOf(rs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After fetching the RS by name, check that the UID matches controllerRef.UID. If it doesn't match, the RS you fetched doesn't actually own the pod; it just has the same name as a previously-deleted RS that owned the pod. In that case, you can return nil, nil since the pod has no RS owner.

if controllerRef.Kind == controllerKindRS.Kind {
rs, err := dc.rsLister.ReplicaSets(pod.Namespace).Get(controllerRef.Name)
if err != nil {
return nil, nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add back the comment that explains why we ignore err: It's because the only possible error returned by the cache is NotFound, and that means there is no RS for this pod.

}
controllerRef := controller.GetControllerOf(rs)
if controllerRef == nil || controllerRef.Kind != controllerKindDep.Kind {
casSlice = append(casSlice, controllerAndScale{rs.UID, *(rs.Spec.Replicas)})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go style suggestion: This line is the end of the "happy path" (everything went as expected - we found an RS). Ideally it should be as unindented as possible, and unexpected things should result in early return statements. This way the "happy path" flows straight down rather than becoming more deeply nested at each step.

As an example, this function could be structured like:

...
if controllerRef == nil {
  return nil, nil
}
if controllerRef.Kind != controllerKindRS.Kind {
  return nil, nil
}
rs, err := ...
if err != nil {
  // The only possible error is NotFound, which is ok here.
  return nil, nil
}
if rs.UID != controllerRef.UID {
  return nil, nil
}
...
if controllerRef != nil && controllerRef.Kind == controllerKindDep.Kind {
  // Skip RS if it's controlled by a Deployment.
  return nil, nil
}
return []controllerAndScale{...}

Notice there's no nesting and everything flows straight down.

ss, err := dc.ssLister.StatefulSets(pod.Namespace).Get(controllerRef.Name)
if err != nil {
return nil, nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check UID after Get.

if controllerRef.Kind == controllerKindSS.Kind {
ss, err := dc.ssLister.StatefulSets(pod.Namespace).Get(controllerRef.Name)
if err != nil {
return nil, nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add back comment about NotFound error being ok.

rc, err := dc.rcLister.ReplicationControllers(pod.Namespace).Get(controllerRef.Name)
if err != nil {
return nil, nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check UID after Get.

add(t, dc.rcStore, rc)
dc.sync(pdbName)

fmt.Println("reeached here 1")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove raw print or convert to t.Log().

dc.sync(pdbName)

// No controllers yet => no disruption allowed
ps.VerifyDisruptionAllowed(t, pdbName, 0)

rc, _ := newReplicationController(t, 1)
rc.Name = "rc 1"
for i := 0; i < podCount; i++ {
updatePodOwnerToRc(t, pods[i], rc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this test be split into multiple test cases, so we don't modify pods that were already added to the pod store? In my experience, that can lead to flaky/racy tests because you are changing things you already gave to the pod store, which it is not designed to support.

If these need to be in one test case/func, is it possible to add new pods rather than changing existing ones?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

took care of all other comments. will try this and see how it works. Not very familiar with this framework

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR LGTM other than the potential flakiness here. If you want, we can just try to measure whether this is going to be flaky and submit it as is if it passes.

One way to do that is something like go test -count 1000 or go test -race -count 1000. You might want to start with a smaller count and work up to what your machine can handle. It seems that Go sometimes runs the tests concurrently and I've locked up my workstation before by putting a count that was too high. :)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go test ./pkg/controller/disruption/... --count 300 --race
race: limit on 8192 simultaneously alive goroutines is exceeded, dying
FAIL k8s.io/kubernetes/pkg/controller/disruption 16.577s

go test ./pkg/controller/disruption/... --count 250 --race
ok k8s.io/kubernetes/pkg/controller/disruption 16.575s

I cant go beyond 250 on my mac. Is there a way to request this count with the prbot ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try running higher counts, but without -race.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go test ./pkg/controller/disruption/... --count 300
ok  	k8s.io/kubernetes/pkg/controller/disruption	2.348s
go test ./pkg/controller/disruption/... --count 400
ok  	k8s.io/kubernetes/pkg/controller/disruption	3.047s
go test ./pkg/controller/disruption/... --count 600
ok  	k8s.io/kubernetes/pkg/controller/disruption	4.573s
go test ./pkg/controller/disruption/... --count 900
ok  	k8s.io/kubernetes/pkg/controller/disruption	6.864s
go test ./pkg/controller/disruption/... --count 1200
ok  	k8s.io/kubernetes/pkg/controller/disruption	9.862s

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Let me know when you've resolved the merge conflicts and I will LGTM.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@enisoc resolved conflicts.

add(t, dc.rcStore, rc)
dc.sync(pdbName)

fmt.Println("reeached here 1, %v", len(pods[0].OwnerReferences))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove or convert to t.Log().

}

func updatePodOwnerToRs(t *testing.T, pod *v1.Pod, rs *extensions.ReplicaSet) {
var controllerKind = apps.SchemeGroupVersion.WithKind("ReplicaSet")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use the vars you already defined in the other file?

@krmayankk krmayankk force-pushed the garbage branch 2 times, most recently from 65c77d6 to e7ac592 Compare May 20, 2017 06:00
controllerKindRS = v1beta1.SchemeGroupVersion.WithKind("ReplicaSet")
controllerKindSS = apps.SchemeGroupVersion.WithKind("StatefulSet")
controllerKindRC = v1.SchemeGroupVersion.WithKind("ReplicationController")
controllerKindDep = v1beta1.SchemeGroupVersion.WithKind("Deployment")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also have Deployments in the apps group. I think you need to use both but I am not fairly certain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, you just compare UID and Kind so it may not be needed after all

}

return cas, nil
casSlice = append(casSlice, controllerAndScale{ss.UID, *(ss.Spec.Replicas)})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it's only one parent that you are returning now, switch all these helpers to return a single object.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kargakis i was hoping to do that change in a subsequent PR, since that would also involve modifying the callers and understanding that logic. Is that ok ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

@enisoc enisoc added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels May 22, 2017
@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 23, 2017
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 24, 2017
@krmayankk
Copy link
Author

@k8s-bot pull-kubernetes-e2e-gce-etcd3 test this

@enisoc
Copy link
Member

enisoc commented May 24, 2017

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 24, 2017
@enisoc
Copy link
Member

enisoc commented May 24, 2017

/assign @derekwaynecarr

@derekwaynecarr
Copy link
Member

/approve

@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, enisoc, krmayankk

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 25, 2017
@enisoc
Copy link
Member

enisoc commented May 25, 2017

@k8s-bot pull-kubernetes-federation-e2e-gce test this

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 45518, 46127, 46146, 45932, 45003)

@k8s-github-robot k8s-github-robot merged commit 749ac27 into kubernetes:master May 25, 2017
k8s-github-robot pushed a commit that referenced this pull request Aug 28, 2017
Automatic merge from submit-queue

simplify disruption controller finder logic

**What this PR does / why we need it**:
Address some comments from #45003 and simplify the PDB controller logic as part of issue #42284

@enisoc @Kargakis @caesarxuchao 

Also it feels like we can get rid of the finders all together since with controller ref, each pod has only controller. Let me know if i should remove that finders all together ?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.