@derekwaynecarr @aveshagarwal help check thanks

looking into this.

@kubernetes/sig-scheduling-bugs

I am able to reproduce this.

I am seeing different behavior with kubectl describe and kubectl get:

Conditions:
  Type			Status		LastHeartbeatTime			LastTransitionTime			Reason			Message
  ----			------		-----------------			------------------			------			-------
  OutOfDisk 		Unknown 	Tue, 21 Mar 2017 09:06:14 -0400 	Tue, 21 Mar 2017 09:06:57 -0400 	NodeStatusUnknown 	Kubelet stopped posting node status.
  MemoryPressure 	Unknown 	Tue, 21 Mar 2017 09:06:14 -0400 	Tue, 21 Mar 2017 09:06:57 -0400 	NodeStatusUnknown 	Kubelet stopped posting node status.
  DiskPressure 		Unknown 	Tue, 21 Mar 2017 09:06:14 -0400 	Tue, 21 Mar 2017 09:06:57 -0400 	NodeStatusUnknown 	Kubelet stopped posting node status.
  Ready 		Unknown 	Tue, 21 Mar 2017 09:06:14 -0400 	Tue, 21 Mar 2017 09:06:57 -0400 	NodeStatusUnknown 	Kubelet stopped posting node status.

_output/bin/kubectl get nodes
NAME              STATUS     AGE       VERSION
192.168.124.120   NotReady   4d        v1.7.0-alpha.0.1399+1e5fa8fed5459a

So kubectl describe shows unkown whereas kubectl get shows NotReady.

@aveshagarwal There also are two other issue need confirm with you if it's by design or not.

1. If there is only one node(all-in-one env), when node become notReady/Unknow, node controller can't add taint to node. multi-nodes no this issue
2. After node controller add taint to node, then manual remove the taint, node controller can't add the notready/unreachable again.

Thanks

@mdshuai first one is by design to avoid mass eviction when all nodes are not ready/unknown. 2nd I am not sure will have to look into that though it seems node controller is honoring user intent so looks good.

Also in my understanding, the original issue here should not affect the end result in a cluster regarding eviction as there is no difference w.r.t action taken irrespective of notready or unreachable taint.

@gmarek I am noticing that when a node goes from NotReady to Unreachable even though node status is unknown I dont see the flow reaching here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go#L530

@aveshagarwal thanks confirm

@mdshuai how many node cluster are you testing on?

@aveshagarwal Three

@mdshuai I think i know what is the issue working on a fix.

@mdshuai fixed it in #43463

The feature is alpha, but what is reported here is indeed a bug for the feature.

No, it's not. It's the desired behavior, as we decided at some point that we want to have only one: 'notReady' or 'unreachable' taint on the Node. Current implementation just keeps the first added taint.

@davidopp @timothysc @kubernetes/sig-scheduling-misc

Oh, I thought we decided that the taint would always match the node condition. So there would only be one taint at a time, but the taint would always reflect whatever the node condition says. That would be more intuitive. How hard would it be to change the behavior to do that?

Easy, but not 1.6 level easy.

I'll explain the logic here.. When we first observe a condition other than 'Ready' we add a node to the queue. Node can spend in queue some time, so we check the condition at the time when Taint is applied, so it's consistent at that time. Indeed if the condition is changed after that we won't reflect that. If we want to change this logic we'd need to decide on how we want to do it - do we want this change to be rate limited together with other taint additions (i.e. reuse the same queue as everything else), or do we want it to modify the taint instantly?

I think that the latter makes more sense, so the proper fix would check the current taint in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go#L686 (and corresponding part for unresponsive part), and if it's different try to update the taint directly. This has the advantage of not accounting as additional "eviction" for the purpose of monitoring.

The proper fix for the former, would be to just 'remove' the key from the queue if the different taint is present in the same line as above.