Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Federation] Daemonset with finalizer not always getting deleted #43828

Closed
csbell opened this issue Mar 29, 2017 · 2 comments
Closed

[Federation] Daemonset with finalizer not always getting deleted #43828

csbell opened this issue Mar 29, 2017 · 2 comments
Assignees
@nikhiljindal
Milestone
next-candidate

Comments

@csbell
Copy link
Contributor

csbell commented Mar 29, 2017

From CI failure here: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2458/build-log.txt

Upon deleting with orphan=true, the finalizer Update() fails and the next time around the federated object is never deleted. This is flaky.

Relevant bits:

I0329 11:50:35.083] I0329 18:49:17.185376       1 daemonset_controller.go:273] Trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.185529       1 daemonset_controller.go:338] Reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.185667       1 daemonset_controller.go:462] Handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 149 2 2017-03-29 18:49:11 +0000 UTC 2017-03-29 18:49:16 +0000 UTC 0xc4229b4398 map[aaa:bbb] map[] [] [federation.kubernetes.io/delete-from-underlying-clusters orphan] } {&LabelSelector{MatchLabels:map[string]string{aaa: bbb,},MatchExpressions:[],} {{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[aaa:bbb] map[] [] [] } {[] [] [{container1 gcr.io/google_containers/serve_hostname:v1.4 [] []  [{ 0 9376 TCP }] [] [] {map[] map[]} [] nil nil nil /dev/termination-log File IfNotPresent nil false false false}] Always 0xc4229b43b0 <nil> ClusterFirst map[]   <nil>  false false false &PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,} []   nil default-scheduler []}} {OnDelete nil} 0 1} {0 0 0 0 0 0 0 0}}
I0329 11:50:35.084] I0329 18:49:17.185966       1 deletion_helper.go:115] Handling deletion of federated dependents for object: federated-daemonset
I0329 11:50:35.084] I0329 18:49:17.186060       1 deletion_helper.go:122] Found finalizer orphan. Nothing to do, just remove the finalizer
I0329 11:50:35.085] E0329 18:49:17.215415       1 daemonset_controller.go:368] Failed to delete federated-daemonset: failed to remove finalizer orphan from daemonset : Operation cannot be fulfilled on daemonsets.extensions "federated-daemonset": the object has been modified; please apply your changes to the latest version and try again
I0329 11:50:35.086] I0329 18:49:17.219857       1 daemonset_controller.go:273] Trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.219936       1 daemonset_controller.go:338] Reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.220059       1 daemonset_controller.go:462] Handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 154 2 2017-03-29 18:49:11 +0000 UTC 2017-03-29 18:49:16 +0000 UTC 0xc422990c58 map[aaa:bbb] map[] [] [orphan] } {&LabelSelector{MatchLabels:map[string]string{aaa: bbb,},MatchExpressions:[],} {{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[aaa:bbb] map[] [] [] } {[] [] [{container1 gcr.io/google_containers/serve_hostname:v1.4 [] []  [{ 0 9376 TCP }] [] [] {map[] map[]} [] nil nil nil /dev/termination-log File IfNotPresent nil false false false}] Always 0xc422990c80 <nil> ClusterFirst map[]   <nil>  false false false &PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,} []   nil default-scheduler []}} {OnDelete nil} 0 1} {0 0 0 0 0 0 0 0}}
I0329 11:50:35.086] I0329 18:49:17.220332       1 deletion_helper.go:115] Handling deletion of federated dependents for object: federated-daemonset
I0329 11:50:35.086] I0329 18:49:17.220420       1 deletion_helper.go:117] obj does not have federation.kubernetes.io/delete-from-underlying-clusters finalizer. Nothing to do
@csbell csbell added this to the next-candidate milestone Mar 29, 2017
@csbell
Copy link
Contributor Author

csbell commented Apr 4, 2017

Discussed this with Nikhil earlier today. The buggy behavior is a result of handling removal of 2 or more annotations. In this case, the daemonset and namespace controllers both try to remove orphan and delete-from finalizers but the code isn't written to expect separate updates.

Seen again here today: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2570

Marking P0. With the fix in, we should be above 90% in no-flakes.

@nikhiljindal nikhiljindal mentioned this issue Apr 5, 2017
Merged
@nikhiljindal
Copy link
Contributor

Sent #44084 to fix this

k8s-github-robot pushed a commit that referenced this issue Apr 6, 2017
Merge pull request #44084 from nikhiljindal/removeFinalizers
586e6d0 
Automatic merge from submit-queue (batch tested with PRs 44084, 42964)

Removing both finalizers in federation controllers in a single update

Fixes #43828

There is a bug right now where the controller fails to delete the object if one finalizer is removed and the second isnt.
This updates the code so that both the finalizers are removed in a single API call. Kept the code changes minimum to enable cherrypick in 1.6.x

cc @csbell @kubernetes/sig-federation-bugs
k8s-github-robot pushed a commit that referenced this issue Apr 11, 2017
Merge pull request #44289 from nikhiljindal/automated-cherry-pick-of-…
9ab768e 
…#44084-upstream-release-1.6

Automatic merge from submit-queue

Automated cherry pick of #44084

Cherry pick of #44084 on release-1.6.

#44084: Removing both finalizers in federation controller in a

```release-note
Fix for [failure to delete federation controllers with finalizers](#43828).
```
mintzhao pushed a commit to mintzhao/kubernetes that referenced this issue Jun 1, 2017
Merge pull request kubernetes#44289 from nikhiljindal/automated-cherr…
e20a677 
…y-pick-of-#44084-upstream-release-1.6

Automatic merge from submit-queue

Automated cherry pick of kubernetes#44084

Cherry pick of kubernetes#44084 on release-1.6.

kubernetes#44084: Removing both finalizers in federation controller in a

```release-note
Fix for [failure to delete federation controllers with finalizers](kubernetes#43828).
```
perotinus pushed a commit to kubernetes-retired/cluster-registry that referenced this issue Sep 2, 2017
Merge pull request #44084 from nikhiljindal/removeFinalizers
0641b60 
Automatic merge from submit-queue (batch tested with PRs 44084, 42964)

Removing both finalizers in federation controllers in a single update

Fixes kubernetes/kubernetes#43828

There is a bug right now where the controller fails to delete the object if one finalizer is removed and the second isnt.
This updates the code so that both the finalizers are removed in a single API call. Kept the code changes minimum to enable cherrypick in 1.6.x

cc @csbell @kubernetes/sig-federation-bugs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikhiljindal nikhiljindal

Labels
None yet
Projects
None yet
Milestone
next-candidate
Development

No branches or pull requests
2 participants
@nikhiljindal @csbell