/sig aws

#62774

Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS

9f62e81

Add aws cloud config:

[global]
disableSecurityGroupIngress = true

/sig cloud-provider

We're also having this issue which makes service.beta.kubernetes.io/aws-load-balancer-security-groups kinda unusable. EnsureLoadBalancer() shouldn't ensure permissions if an SG is a custom one here https://github.com/kubernetes/kubernetes/blob/release-1.13/pkg/cloudprovider/providers/aws/aws.go#L3579 .

@Raffo could you please take a look?

This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the service.beta.kubernetes.io/aws-load-balancer-security-groups should ensure it's attached to the ELB/LB, and remain untouched.

Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.

@uty unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the sig-aws meetings.

Suppose you have an existing EKS cluster, gateways, the whole nine yards...

Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs.

You expect, by using this SG id (sg-133780085) in conjunction with this service annotation that:

• All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).
• All entries in your custom security group are intact, and would be retained by the control plane.

Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules.

Temporary solution:

Run the override script (in our case, assign the serviceAnnotations to the gateway).
Still use a custom security group, but leave it empty.
When all and done, add your custom entries to the SG.

As of 08/22, this cheap workaround, works. Still needs investigating.

Suppose you have an existing EKS cluster, gateways, the whole nine yards...

Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs.

You expect, by using this SG id (sg-133780085) in conjunction with this service annotation that:

• All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).
• All entries in your custom security group are intact, and would be retained by the control plane.

Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules.

Temporary solution:

Run the override script (in our case, assign the serviceAnnotations to the gateway).
Still use a custom security group, but leave it empty.
When all and done, add your custom entries to the SG.

As of 08/22, this cheap workaround, works. Still needs investigating.

Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.

Hi,

It looks I have the same issue. Company policy dictates that all SG's have a specific name.
Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine.

I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with
service.beta.kubernetes.io/aws-load-balancer-security-groups: "{elb-security-group-id}"
Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one.
I added the a file with contents

[global]
disableSecurityGroupIngress = true

in the /etc/cloud/cloud.cfg.d location.

But I cannot get it to work.
We use as
service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0", which probably creates this extra rule.

I cannot find ( or am probably not able to read correctly) the correct config.
Should the aws-load-balancer-internal be used or not? Do I need another directive?

Maybe someone can help me out with some pointers?

Thanks in advance.

It's possible to work around this by setting loadBalancerSourceRanges to something bogus (i.e. 127.0.0.1/32) and putting your desired SG in the service.beta.kubernetes.io/aws-load-balancer-extra-security-groups annotation instead (k8s won't modify this one).

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale