Skip to content

node not exist failure during node status update flush controller's log #30898

@jingxu97

Description

@jingxu97

@saad-ali @matchstick
From issue #29903, I noticed that under attachdetach controller, reconciler keeps updating node status every 100ms. In some cases, the node no longer exists but reconciler still has a stale cache information of it. The error message quickly flush the kube-controller-manager's log.

  1. Quick fix: Lower the level of this log so by default it won't show up
  2. Right fix: have a good comprehensive understanding of all scenarios especially when certain components in the system fail/exit, how the volume manager should react/recover from them.

Related code and PR: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go#L104
#29358
#30737

Metadata

Metadata

Assignees

Labels

area/kubeletpriority/backlogHigher priority than priority/awaiting-more-evidence.sig/storageCategorizes an issue or PR as relevant to SIG Storage.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions