Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.5

Environment: Any juju deployed environment

• Cloud provider or hardware configuration: Any cloud used by juju
• OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS
• Kernel (e.g. uname -a): 4.4.0-66-generic
• Install tools:
• Others:

What happened:
Removed all kubernetes-master units and then re-added one master node and that master was failing.

What you expected to happen:
The master would function properly.

How to reproduce it (as minimally and precisely as possible):
Remove all kubernetes-master units and then re-added one master. There is also this tests that shows the issue: https://github.com/ktsakalozos/bundle-canonical-kubernetes/blob/bug/master-check/tests/40-remove-add-master-check.py

Anything else we need to know:
This is for the deployments done with juju