Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove setInitError. #59020

Merged
merged 1 commit into from
Jan 30, 2018
Merged

Remove setInitError. #59020

merged 1 commit into from
Jan 30, 2018
+1 −12

Conversation

brendandburns
Copy link
Contributor

What this PR does / why we need it:
Removes setInitError, it's not sure it was ever really used, and it causes the kubelet to hang and get wedged.

Which issue(s) this PR fixes
Fixes #46086

Special notes for your reviewer:
If initializeModules() in kubelet.go encounters an error, it calls runtimeState.setInitError(...)

kubernetes/pkg/kubelet/kubelet.go

Line 1339 in 47d61ef

kl.runtimeState.setInitError(err)

The trouble with this is that initError is never cleared, which means that runtimeState.runtimeErrors() always returns this initError, and thus pods never start sync-ing.

In normal operation, this is expected and desired because eventually the runtime is expected to become healthy, but in this case, initError is never updated, and so the system just gets wedged.

kubernetes/pkg/kubelet/kubelet.go

Line 1751 in 47d61ef

if rs := kl.runtimeState.runtimeErrors(); len(rs) != 0 {

We could add some retry to initializeModules() but that seems unnecessary, as eventually we'd want to just die anyway. Instead, just log fatal and die, a supervisor will restart us.

Note, I'm happy to add some retry here too, if that makes reviewers happier.

Release note:

Prevent kubelet from getting wedged if initialization of modules returns an error.

@feiskyer @dchen1107 @janetkuo

@kubernetes/sig-node-bugs

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 30, 2018
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 30, 2018
feiskyer
feiskyer approved these changes Jan 30, 2018
View reviewed changes
Copy link
Member

@feiskyer feiskyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yujuhong
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 30, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brendandburns, yujuhong

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit a18f086 into kubernetes:master Jan 30, 2018
This was referenced Feb 1, 2018
Merged
Merged
Merged
@k8s-cherrypick-bot
Copy link

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

@jdumars jdumars added this to the v1.9 milestone Feb 1, 2018
@jdumars
Copy link
Member

jdumars commented Feb 1, 2018

@mbohlool @jpbetz @wojtek-t this needs to go back to 1.7
see:
1.9: #59162
1.8: #59163
1.7: #59164

k8s-github-robot pushed a commit that referenced this pull request Feb 2, 2018
Merge pull request #59163 from brendandburns/automated-cherry-pick-of…
7e29ffe 
…-#59020-upstream-release-1.8

Automatic merge from submit-queue.

Automated cherry pick of #59020: Remove setInitError.

Cherry pick of #59020 on release-1.8.

#59020: Remove setInitError.
k8s-github-robot pushed a commit that referenced this pull request Feb 2, 2018
Merge pull request #59162 from brendandburns/automated-cherry-pick-of…
e68f37f 
…-#59020-upstream-release-1.9

Automatic merge from submit-queue.

Automated cherry pick of #59020: Remove setInitError.

Cherry pick of #59020 on release-1.9.

#59020: Remove setInitError.
@k8s-cherrypick-bot
Copy link

Commit found in the "release-1.9" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

k8s-github-robot pushed a commit that referenced this pull request Feb 3, 2018
Merge pull request #59164 from brendandburns/automated-cherry-pick-of…
2be5391 
…-#59020-upstream-release-1.7

Automatic merge from submit-queue.

Automated cherry pick of #59020: Remove setInitError.

Cherry pick of #59020 on release-1.7.

#59020: Remove setInitError.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feiskyer feiskyer feiskyer approved these changes

@dashpole dashpole Awaiting requested review from dashpole

@sjpotter sjpotter Awaiting requested review from sjpotter

Assignees

@yujuhong yujuhong

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.9
Development

Successfully merging this pull request may close these issues.

kubelet gets in an endless loop when initializeModules failed
7 participants
@brendandburns @yujuhong @k8s-ci-robot @k8s-github-robot @k8s-cherrypick-bot @jdumars @feiskyer