Run a kubeadm init with 'cloud-provider: external' set and join a worker to the cluster. Kube-Proxy will schedule and run on all nodes even with the uninitialised taints in place

if this cannot be fixed by adjusting the kubeadm kube-proxy manifest and config map:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/proxy/manifests.go

or by adjusting the kubeproxy config that the kubeadm config embeds:
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/apis/kubeproxyconfig/types.go#L100

then it's hard to qualify this as a kubeadm issue.

Thanks. I'll have a play with the DS and if I can't do anything there I'll kick it upstairs.

Running the kube-proxy regardless of taints via the - Exists line (re-introduced in 8dcb980) means that kube-proxy runs before the NodeIPs are initialised by the cloud-provider. That breaks kube-proxy.
kube-proxy has to respect the 'node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule' taint put in place by kubelet.

@NeilW
kubernetes/kubernetes#65931
it fixed one thing but broke another....we did similar with the last changes in that file before that.

we are really back and forth on this one and to be honest i don't know what's best here in terms of rules, except that i know for sure that we need to expose those hardcoded addon configurations to the users and let them adjust the values they want.

/assign @timothysc
/assign @luxas

It looks like an architectural issue that's dropping down the gap.

It's unfortunate that TolerationOperator only offers the "Exists" and "Equal" predicates, and not the "NotIn" predicate understood by node selectors. With "NotIn," we could express "everything except this whitelist of taints"—such as the cloud provider one mentioned above.

/cc @kubernetes/sig-scheduling-bugs

I believe this issue happened at the time that kube-proxy (which is a Daemon) was being scheduled by DaemonSet controller. I wonder if the same issue exists now (k8s 1.12+) when Daemons are scheduled by the default scheduler.

Same problem

$ kubectl -n kube-system logs kube-proxy-6hw29
W1012 14:36:33.306545       1 server.go:609] Failed to retrieve node IP: host IP unknown; known addresses: []
I1012 14:36:33.306605       1 server_others.go:189] Using ipvs Proxier.
W1012 14:36:33.307482       1 proxier.go:328] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I1012 14:36:33.307639       1 server_others.go:216] Tearing down inactive rules.
I1012 14:36:33.337372       1 server.go:447] Version: v1.12.1
I1012 14:36:33.348285       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I1012 14:36:33.348458       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1012 14:36:33.348861       1 conntrack.go:83] Setting conntrack hashsize to 32768
I1012 14:36:33.359809       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1012 14:36:33.360044       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1012 14:36:33.360491       1 config.go:102] Starting endpoints config controller
I1012 14:36:33.360598       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I1012 14:36:33.360724       1 config.go:202] Starting service config controller
I1012 14:36:33.361784       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I1012 14:36:33.460899       1 controller_utils.go:1034] Caches are synced for endpoints config controller
I1012 14:36:33.462227       1 controller_utils.go:1034] Caches are synced for service config controller

The patch I use to workaround the issue is

kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": { "tolerations": [ { "key": "CriticalAddonsOnly", "operator": "Exists" }, { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" } ] } } } }'

Has anyone considered adding a "daemonTaints" field to kubeadm's ClusterConfiguration type that would allow operators to define locally-known taints that all system daemons like kube-proxy should tolerate? An example from my world: "nvidia.com/gpu" with the "NoSchedule" effect. I don't expect kubeadm to know about that, but it would be nice if there was a common, structured way to tell it about that.

so one of the reasons we didn't expose full control of the addon manifests in the latest iteration of the config was because we then lose some of the control when we do upgrades.

so by planning a field such as "daemonTaints" we need to evaluate upgrade scenarios too.

overall this falls in the the bucket of customization the we currently simply do not allow and users need to patch.

this idea here is mostly outdated (and not approved) but it spawned a bit of a discussion about the problems at hand: #1091

Thanks for the background.

Instead of "daemonTaints," I should have said "daemonTolerations," but you get the idea.

@timothysc I'm interested in helping with this issue or taking it as I understand it is related to the broader add-on issue as @neolit123 suggested.

here are some key points in this thread:
#1027 (comment)
#1027 (comment)
#1027 (comment)

this isn't really a kubeadm bug, because in kubernetes/kubernetes#65931 we arguably fixed a bigger problem. but with that PR we introduced this problem, which then sort of transitions into a feature request in the scheduler as outlined here #1027 (comment)

i don't think we can do much in this ticket for this cycle in terms of code.
what we can do is document the workaround here:
#1027 (comment)

if anyone wants to help this is the place:
https://github.com/kubernetes/website/blob/master/content/en/docs/setup/independent/troubleshooting-kubeadm.md

/kind documentation

i will send a website PR to add a troubleshooting note.

sent docs PR to document the workaround:
kubernetes/website#13033

moving this to the Next milestone.
we need to figure out if this is still a problem in latest 1.13 and 1.14.

Unfortunately it is. If I remove the toleration patch I get

$ kubectl -n 'kube-system' logs kube-proxy-m4ssh
W0308 10:27:28.506237       1 node.go:108] Failed to retrieve node IP: host IP unknown; known addresses: []
I0308 10:27:28.506517       1 server_others.go:189] Using ipvs Proxier.
W0308 10:27:28.507144       1 proxier.go:366] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I0308 10:27:28.509332       1 server_others.go:216] Tearing down inactive rules.
I0308 10:27:28.540195       1 server.go:483] Version: v1.13.4

when building a new cluster with terraform.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

docs PR merged, but not much we can do here on the kubeadm side to conform this use case.
see #1027 (comment)
/close

@neolit123: Closing this issue.