Currently, the BFF is visible to the outside world. I've done some preliminary investigations here and the problem seems to be related to how the service annotations interact with the security groups.

Our intention was that any endpoint only be available within our network. So, we:

1. Define a Security Group that only allows incoming traffic from the configured prefix list
2. Attach the Security Group to the BFF service resource definition.
3. Override the prometheus operator values and attach the Security Group to the Grafana service resource definition.

This approach still doesn't work due to an issue in Kubernetes, which is addressed in a (as of this writing) yet to be merged PR.

Essentially, if we specify the Security Group (SG) with the service.beta.kubernetes.io/aws-load-balancer-security-groups annotation, the rules get overwritten with the kubernetes default SG (Allow from 0.0.0.0/0). If we assign our SG via the service.beta.kubernetes.io/aws-load-balancer-extra-security-groups annotation, it remains unchanged, but Kubernetes still creates the default SG. While both SGs are assigned to the ELB. According to the AWS documentation:

If there is more than one rule for a specific port, we apply the most permissive rule.
For example, if you have a rule that allows access to TCP port 22 (SSH)
from IP address 203.0.113.1  and another rule that allows access to
TCP port 22 from everyone, everyone has access to TCP port 22.

Suggested Solution

We need to figure out what the ip ranges for the plist are and apply the service.beta.kubernetes.io/load-balancer-source-ranges annotation (and remove the other annotations), at least until the PR mentioned above gets merged.