Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authentication Issues On EKS Cluster with Fargate Policy #1092

Closed
joshkurz opened this issue Dec 3, 2019 · 2 comments
Closed

Authentication Issues On EKS Cluster with Fargate Policy #1092

joshkurz opened this issue Dec 3, 2019 · 2 comments

Comments

@joshkurz
Copy link

joshkurz commented Dec 3, 2019 •
edited
Loading

Trying to get the alb-ingress-controller running on a fargate node in EKS. I have the application spun up and running correctly on fargate, however It seems the pods are not able to reach the correct aws apis to make the necessary calls to build the ALBs.

Here are some error logs I am getting.

kubectl logs -f alb-ingress-controller-69f8785c5c-rgdc2 -n kube-system
W1203 22:14:41.492118       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.1.3
  Build:      git-6101b02d
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

I1203 22:14:41.559863       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I1203 22:14:41.560233       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1203 22:14:41.560371       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I1203 22:14:41.560616       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1203 22:14:41.560655       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I1203 22:14:41.560933       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I1203 22:14:41.561302       1 controller.go:121] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I1203 22:14:41.561687       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I1203 22:14:41.578204       1 leaderelection.go:214] successfully acquired lease kube-system/ingress-controller-leader-alb
I1203 22:14:41.678801       1 controller.go:134] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I1203 22:14:41.779075       1 controller.go:154] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1
E1203 22:16:18.113958       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:18:02.013655       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:19:45.213243       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:21:21.090416       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:23:04.983276       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:24:48.372362       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:26:34.078393       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}
E1203 22:28:10.508719       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"  "controller"="alb-ingress-controller" "request"={"Namespace":"jkurz","Name":"jkurz-ingress"}

I have the fargate-policy using a role that allows it to execute fargate application on EKS, and it also has the custom IAM policy ALBIngressControllerIAMPolicy that I got from running this command.

aws iam attach-role-policy \
--policy-arn arn:aws:iam::111122223333:policy/ALBIngressControllerIAMPolicy \
--role-name fargateExecutionRoleName

Is there anything I am missing to get this to work? is it possible to run the alb-controller on a fargate node in EKS? Just want to be sure before I continue. Thanks
@M00nF1sh
Copy link
Collaborator

M00nF1sh commented Dec 3, 2019 •
edited
Loading

@joshkurz
The fargateExecutionRole is actually the role for the kubelet(and kube-proxy) that running your fargate pod, thus it's not the role for fargate pod(thus our alb-ingress-controller).
To run alb-ingress-controller as a fargate pod, you need to use iam-for-pods, or simply add AWS_ACCESS_KEY_ID as environment variable if it just for testing purpose.
BTW, only v1.1.4 with mode IP works for fargate pods: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v1.1.4, detailed instructions are available on this release page

@joshkurz
Copy link
Author

joshkurz commented Dec 4, 2019

Ok thanks for the info.

@joshkurz joshkurz closed this as completed Dec 4, 2019
@jpadams jpadams mentioned this issue Dec 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joshkurz @M00nF1sh