Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

infra/upload-config/kubeadm phase fails saying creating config-maps is forbidden with user "Kubernetes-admin" #3087

Closed
teja-rafay opened this issue Jul 5, 2024 · 15 comments
Closed

infra/upload-config/kubeadm phase fails saying creating config-maps is forbidden with user "Kubernetes-admin" #3087

teja-rafay opened this issue Jul 5, 2024 · 15 comments
Labels
area/phases kind/support Categorizes issue or PR as a support question.

Comments

@teja-rafay
Copy link

What happened?

while I ran kubeadm init with addition to someother code of mine to bring up a Kubernetes cluster it fails with the below error
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

What did you expect to happen?

The config map should be created without any issues. and the init run should be done successfully.

How can we reproduce it (as minimally and precisely as possible)?

I wrote a go code calling the kubeadm init functions in it. when I start to initialise my cluster components it fails at the upload config phase

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2

Cloud provider

it's a bare metal node.

OS version

On Linux:

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
$ uname -a
Linux ip-172-31-6-57.us-west-2.compute.internal 3.10.0-1160.76.1.el7.x86_64 kubernetes/kubernetes#1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

containers version 1.7.18

Related plugins (CNI, CSI, ...) and versions (if applicable)

cni-plugins-linux-amd64-v1.5.1
@teja-rafay teja-rafay added the kind/bug Categorizes issue or PR as related to a bug. label Jul 5, 2024
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 5, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jul 5, 2024
@HirazawaUi
Copy link

/sig cluster-lifecycle
/area kubeadm

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. area/kubeadm and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 5, 2024
@neolit123
Copy link
Member

/transfer kubeadm

@k8s-ci-robot k8s-ci-robot transferred this issue from kubernetes/kubernetes Jul 6, 2024
@neolit123 neolit123 added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. and removed sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. area/kubeadm labels Jul 6, 2024
@neolit123
Copy link
Member

while I ran kubeadm init with addition to someother code of mine to bring up a Kubernetes cluster it fails with the below error
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

there was a migration in kubeadm 1.29 to have separate admin.conf and super-admin.conf
#2414

you can read the design here:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/4214-separate-super-user-kubeconfig

probably your admin.conf which should be part of the kubeadm:cluster-admins group is missing a clusterrolebinding to the cluster-admin role. try

kubectl create clusterrolebinding kubeadm:cluster-admins --clusterrole=cluster-admin --group=kubeadm:cluster-admins

your cluster was not migrated properly and maybe you did not call kubeadm upgrade; you should share more details on how you call commands exactly and what is the setup.

@teja-rafay
Copy link
Author

Just to add. Mine is not an upgrade cluster. It's a standalone one at v1.30.2. My code is pretty much the same on the kubernetes level as kubeadm. Some extra things we have are come of our storage applications and other things. Let me know if you need anything else

@neolit123
Copy link
Member

apply the fix i mentioned in the previous comment.

@neolit123
Copy link
Member

are you calling other kubeadm phases or only upload-config?

@teja-rafay
Copy link
Author

teja-rafay commented Jul 6, 2024 •
edited
Loading

Almost ever single one of them. Any particular one you want me to check in particular?

@neolit123
Copy link
Member

  • i need to see the exact order of phases
  • show output of the upload-config phase with the flag --v=5 added

@neolit123
Copy link
Member

here is the default order of phases:

preflight                     Run pre-flight checks
certs                         Certificate generation
  /ca                           Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
  /apiserver                    Generate the certificate for serving the Kubernetes API
  /apiserver-kubelet-client     Generate the certificate for the API server to connect to kubelet
  /front-proxy-ca               Generate the self-signed CA to provision identities for front proxy
  /front-proxy-client           Generate the certificate for the front proxy client
  /etcd-ca                      Generate the self-signed CA to provision identities for etcd
  /etcd-server                  Generate the certificate for serving etcd
  /etcd-peer                    Generate the certificate for etcd nodes to communicate with each other
  /etcd-healthcheck-client      Generate the certificate for liveness probes to healthcheck etcd
  /apiserver-etcd-client        Generate the certificate the apiserver uses to access etcd
  /sa                           Generate a private key for signing service account tokens along with its public key
kubeconfig                    Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  /admin                        Generate a kubeconfig file for the admin to use and for kubeadm itself
  /super-admin                  Generate a kubeconfig file for the super-admin
  /kubelet                      Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  /controller-manager           Generate a kubeconfig file for the controller manager to use
  /scheduler                    Generate a kubeconfig file for the scheduler to use
etcd                          Generate static Pod manifest file for local etcd
  /local                        Generate the static Pod manifest file for a local, single-node local etcd instance
control-plane                 Generate all static Pod manifest files necessary to establish the control plane
  /apiserver                    Generates the kube-apiserver static Pod manifest
  /controller-manager           Generates the kube-controller-manager static Pod manifest
  /scheduler                    Generates the kube-scheduler static Pod manifest
kubelet-start                 Write kubelet settings and (re)start the kubelet
upload-config                 Upload the kubeadm and kubelet configuration to a ConfigMap
  /kubeadm                      Upload the kubeadm ClusterConfiguration to a ConfigMap
  /kubelet                      Upload the kubelet component config to a ConfigMap
upload-certs                  Upload certificates to kubeadm-certs
mark-control-plane            Mark a node as a control-plane
bootstrap-token               Generates bootstrap tokens used to join a node to a cluster
kubelet-finalize              Updates settings relevant to the kubelet after TLS bootstrap
  /enable-client-cert-rotation  Enable kubelet client certificate rotation
  /experimental-cert-rotation   Enable kubelet client certificate rotation (DEPRECATED: use 'enable-client-cert-rotation' instead)
addon                         Install required addons for passing conformance tests
  /coredns                      Install the CoreDNS addon to a Kubernetes cluster
  /kube-proxy                   Install the kube-proxy addon to a Kubernetes cluster
show-join-command             Show the join command for control-plane and worker node

for upload-config to work, there are a number of things that must be in place:

  • a CA cert/key on disk
  • a working kube-apiserver
  • a valid super-admin.conf / admin.conf on disk.

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace error execution phase infra/upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: configmaps is forbidden: User "kubernetes-admin" cannot create resource "configmaps" in API group "" in the namespace "kube-system" To see the stack trace of this error execute with --v=5 or higher

the CRB should have been created in an earlier phase already.

@teja-rafay
Copy link
Author

these are the order of phases that involve kubeadm

                        kubeadmPhases.NewCertsPhase(),
                        kubeadmPhases.NewKubeConfigPhase(),
			kubeadmPhases.NewKubeletStartPhase(),
			kubeadmPhases.NewControlPlanePhase(),
			kubeadmPhases.NewEtcdPhase(),
			kubeadmPhases.NewWaitControlPlanePhase(),
			kubeadmPhases.NewUploadConfigPhase(),
			kubeadmPhases.NewUploadCertsPhase(),
			kubeadmPhases.NewMarkControlPlanePhase(),
			kubeadmPhases.NewBootstrapTokenPhase(),
			kubeadmPhases.NewKubeletFinalizePhase(),
			kubeadmPhases.NewAddonPhase(),

@neolit123
Copy link
Member

neolit123 commented Jul 6, 2024 •
edited
Loading

are you calling kubeadm source code in some sort of a forked way in your project?
if so you are missing:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/init.go#L516-L545

d.client, err = kubeconfigphase.EnsureAdminClusterRoleBinding(kubeadmconstants.KubernetesDir, nil)
if err != nil {
	return nil, errors.Wrapf(err, "could not bootstrap the admin user in file %s", kubeadmconstants.AdminKubeConfigFileName)
}

@teja-rafay
Copy link
Author

Yes @neolit123 let me try adding the missing part to my code. I'll get back to you once I test it. Thanks

@neolit123
Copy link
Member

@teja-rafay

it should work.
note, we don't provide support for users forking or embedding parts of kubeadm source code.
we only provide support for users running the binary.

@neolit123 neolit123 added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Jul 6, 2024
@teja-rafay
Copy link
Author

Thanks @neolit123 it's working🙌🏻
Understood. will keep that in mind

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/phases kind/support Categorizes issue or PR as a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@neolit123 @k8s-ci-robot @HirazawaUi @teja-rafay