Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup orphan logging that goes on in the sync loop. #44938

Merged
merged 1 commit into from
Apr 28, 2017
Merged

Cleanup orphan logging that goes on in the sync loop. #44938

merged 1 commit into from
Apr 28, 2017

Conversation

jayunit100
Copy link
Member

@jayunit100 jayunit100 commented Apr 26, 2017 •
edited
Loading

What this PR does / why we need it:

Fixes #44937

Before this PR The older logs were like this:

E0426 00:06:33.763347   21247 kubelet_volumes.go:114] Orphaned pod "35c4a858-2a12-11e7-910c-42010af00003" found, but volume paths are still present on disk.
E0426 00:06:33.763400   21247 kubelet_volumes.go:114] Orphaned pod "e7676365-1580-11e7-8c27-42010af00003" found, but volume paths are still present on disk.

The problem being that, all the volumes were spammed w/ no summary info.

After this PR the logs look like this:

E0426 01:32:27.295568   22261 kubelet_volumes.go:129] Orphaned pod "408b060e-2a1d-11e7-90e8-42010af00003" found, but volume paths are still present on disk. : There were a total of 2 errors similar to this.  Turn up verbosity to see them.
E0426 01:32:29.295515   22261 kubelet_volumes.go:129] Orphaned pod "408b060e-2a1d-11e7-90e8-42010af00003" found, but volume paths are still present on disk. : There were a total of 2 errors similar to this.  Turn up verbosity to see them.
E0426 01:32:31.293180   22261 kubelet_volumes.go:129] Orphaned pod "408b060e-2a1d-11e7-90e8-42010af00003" found, but volume paths are still present on disk. : There were a total of 2 errors similar to this.  Turn up verbosity to see them.

And with logging turned up, the extra info logs are shown with details:

E0426 01:34:21.933983   26010 kubelet_volumes.go:129] Orphaned pod "1c565800-2a20-11e7-bbc2-42010af00003" found, but volume paths are still present on disk. : There were a total of 3 errors similar to this.  Turn up verbosity to see them.
I0426 01:34:21.934010   26010 kubelet_volumes.go:131] Orphan pod: Orphaned pod "1c565800-2a20-11e7-bbc2-42010af00003" found, but volume paths are still present on disk.
I0426 01:34:21.934015   26010 kubelet_volumes.go:131] Orphan pod: Orphaned pod "408b060e-2a1d-11e7-90e8-42010af00003" found, but volume paths are still present on disk.
I0426 01:34:21.934019   26010 kubelet_volumes.go:131] Orphan pod: Orphaned pod "e7676365-1580-11e7-8c27-42010af00003" found, but volume paths are still present on disk.

Release note

Roll up volume error messages in the kubelet sync loop.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 26, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@k8s-github-robot k8s-github-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. release-note-label-needed labels Apr 26, 2017
@jayunit100
Copy link
Member Author

part of general kubelet status / health improvements issue #45029

@jayunit100
Copy link
Member Author

cc @kubernetes/sig-node-pr-reviews

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Apr 28, 2017
@yujuhong
Copy link
Contributor

/lgtm

/cc @kubernetes/sig-storage-pr-reviews, as this may affect the debuggability(?).

@k8s-ci-robot k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Apr 28, 2017
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 28, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jayunit100, yujuhong

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. labels Apr 28, 2017
derekwaynecarr
derekwaynecarr reviewed Apr 28, 2017
View reviewed changes
pkg/kubelet/kubelet_volumes.go
@@ -107,18 +110,29 @@ func (kl *Kubelet) cleanupOrphanedPodDirs(
// If there are still volume directories, do not delete directory
volumePaths, err := kl.getPodVolumePathListFromDisk(uid)
if err != nil {
glog.Errorf("Orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err)
orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jayunit100 do you have any insight re: https://bugzilla.redhat.com/show_bug.cgi?id=1445410 -- i find it strange we hit times where err printed nil here ;-)

Copy link
Member Author

@jayunit100 jayunit100 Apr 28, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think a while back it also used to spam this if there were any volume paths floating around 041fa64#diff-79d1ea3801b52a2add9229183049f4c2

this is fixed now, so it might be lagging in some older forks/distros ., i doubt you will ever see <nil> err running off of the current or any recent (4 months) master fork

@jayunit100
Copy link
Member Author

@yujuhong why does it hate my release note?

@yujuhong
Copy link
Contributor

@yujuhong why does it hate my release note?

My guess is that you need to use the "```release-note" format

@ncdc
Copy link
Member

ncdc commented Apr 28, 2017

/release-note

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Apr 28, 2017
@ncdc ncdc removed the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Apr 28, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 45033, 44961, 45021, 45097, 44938)

@k8s-github-robot k8s-github-robot merged commit e06fc08 into kubernetes:master Apr 28, 2017
@guillelb
Copy link

guillelb commented Oct 20, 2017 •
edited
Loading

I still have this problem in version 1.7.8:

Oct 20 12:44:42 k8svam2x4 kubelet[26338]: E1020 12:44:42.142596   26338 kubelet_volumes.go:128] Orphaned pod "00d6cef6-b244-11e7-9d62-000c29ee2ee0" found, but volume paths are still present on disk. : There were a total of 28 errors similar to this.  Turn up verbosity to see them.
Oct 20 12:44:44 k8svam2x4 kubelet[26338]: E1020 12:44:44.138140   26338 kubelet_volumes.go:128] Orphaned pod "00d6cef6-b244-11e7-9d62-000c29ee2ee0" found, but volume paths are still present on disk. : There were a total of 28 errors similar to this.  Turn up verbosity to see them.

Which version will solve this?

@CodeJjang
Copy link

Still suffering from this in 1.8.5(-rancher1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@derekwaynecarr derekwaynecarr derekwaynecarr left review comments

Assignees

@resouer resouer

@tmrts tmrts

@yujuhong yujuhong

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.