Skip to content

fix mixed protocol issue for azure load balancer #74200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 22, 2019
Merged

fix mixed protocol issue for azure load balancer #74200

merged 1 commit into from
Feb 22, 2019

Conversation

andyzhangx
Copy link
Member

@andyzhangx andyzhangx commented Feb 18, 2019 •
edited
Loading

What type of PR is this?
/kind bug

What this PR does / why we need it:
fix mixed protocol issue for azure load balancer, with below config (service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"), azure provider will create both TCP and UDP rules for the service.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"
  name: web
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  sessionAffinity: None
  type: LoadBalancer

With this PR, you could see below both TCP and UDP rules are created for the service:
image

Which issue(s) this PR fixes:

Fixes #73849

Special notes for your reviewer:
Original PR(#67986) is not completed, I have no idea why I submitted a non-completed PR at that time...

Does this PR introduce a user-facing change?:

fix mixed protocol issue for azure load balancer

/kind bug
/assign @feiskyer
/priority important-soon
/sig azure

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 18, 2019
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/azure cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 18, 2019
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. labels Feb 18, 2019
@justaugustus
Copy link
Member

/test pull-kubernetes-e2e-gce

@feiskyer
Copy link
Member

/test pull-kubernetes-e2e-aks-engine-azure

feiskyer
feiskyer reviewed Feb 19, 2019
View reviewed changes
Copy link
Member

@feiskyer feiskyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes LGTM, but the tests have failed. Let's fix the test and run again.

@andyzhangx
Copy link
Member Author

@ritazh are you fixing this pull-kubernetes-e2e-aks-engine-azure testing issue?

W0219 01:54:20.056] 2019/02/19 01:54:20 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 7.243855407s
W0219 01:54:20.057] 2019/02/19 01:54:20 main.go:297: Something went wrong: failed to acquire k8s binaries: error starting /go/src/k8s.io/release/push-build.sh --nomock --verbose --noupdatelatest --bucket=kubernetes-release-pull --ci --gcs-suffix=/pull-kubernetes-e2e-aks-engine-azure --allow-dup: fork/exec /go/src/k8s.io/release/push-build.sh: no such file or directory
W0219 01:54:20.060] Traceback (most recent call last):
W0219 01:54:20.060]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 764, in <module>
W0219 01:54:20.090]     main(parse_args())
W0219 01:54:20.090]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 615, in main
W0219 01:54:20.091]     mode.start(runner_args)
W0219 01:54:20.091]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start
W0219 01:54:20.091]     check_env(env, self.command, *args)
W0219 01:54:20.091]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0219 01:54:20.091]     subprocess.check_call(cmd, env=env)
W0219 01:54:20.091]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W0219 01:54:20.129]     raise CalledProcessError(retcode, cmd)

@feiskyer
Copy link
Member

Refer kubernetes/test-infra#11355

@ritazh
Copy link
Member

ritazh commented Feb 19, 2019

/test pull-kubernetes-e2e-aks-engine-azure

@justaugustus
Copy link
Member

@andyzhangx @ritazh --

W0219 19:59:34.733] 2019/02/19 19:59:34 main.go:297: Something went wrong: starting e2e cluster: error building hyperkube error reading docker passoword file : open : no such file or directory.

@ritazh
Copy link
Member

ritazh commented Feb 20, 2019

Here is a fix for that issue: kubernetes/test-infra#11378

@feiskyer
Copy link
Member

/milestone 1.14

@k8s-ci-robot
Copy link
Contributor

@feiskyer: The provided milestone is not valid for this repository. Milestones in this repository: [next-candidate, v1.10, v1.11, v1.12, v1.13, v1.14, v1.15, v1.16, v1.17, v1.18, v1.4, v1.5, v1.6, v1.7, v1.8, v1.9]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@feiskyer
Copy link
Member

/milestone v1.14

@k8s-ci-robot k8s-ci-robot added this to the v1.14 milestone Feb 20, 2019
@andyzhangx
Copy link
Member Author

/test pull-kubernetes-e2e-aks-engine-azure

@ritazh
Copy link
Member

ritazh commented Feb 20, 2019

@andyzhangx the job won't work yet until a new image of kubekins-e2e is built and we need to update the job config after that

@andyzhangx
Copy link
Member Author

/test pull-kubernetes-e2e-aks-engine-azure

@andyzhangx
Copy link
Member Author

@ritazh the test error is different now, only 2 failed / 224 succeeded

test/e2e/framework/framework.go:704
Timed out after 60.000s.
Expected
    <*errors.errorString | 0xc0016eef10>: {
        s: "expected state to be terminated. Got pod status: {Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.240.0.65 PodIP: StartTime:2019-02-21 06:49:04 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041 State:{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:docker.io/library/busybox:1.29 ImageID: ContainerID:}] QOSClass:BestEffort}",
    }
to be nil
test/e2e/common/kubelet.go:123

@ritazh
Copy link
Member

ritazh commented Feb 21, 2019 •
edited
Loading

yes we are seeing flaky test failures

@feiskyer
Copy link
Member

Thanks @ritazh @andyzhangx. The test failures are actually not related with this PR. Let's get this in first.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 22, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit eae5b73 into kubernetes:master Feb 22, 2019
This was referenced Feb 25, 2019
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Feb 26, 2019
@k8s-ci-robot
Merge pull request #74492 from andyzhangx/automated-cherry-pick-of-#7…
20ff821 
…4200-upstream-release-1.13

Automated cherry pick of #74200: add mixed protocol support for azure load balancer
k8s-ci-robot added a commit that referenced this pull request Mar 7, 2019
@k8s-ci-robot
Merge pull request #74494 from andyzhangx/automated-cherry-pick-of-#7…
a0ed84f 
…4200-upstream-release-1.12

Automated cherry pick of #74200: add mixed protocol support for azure load balancer
@hargrave81
Copy link

I'm a n00b, how do i get this feature to work with my AKS?

@andyzhangx
Copy link
Member Author

@hargrave81
This PR is cherry picked into following k8s versions:

v1.12.7
v1.13.4
v1.14.0

And AKS v1.12.7 is now available, you could try using the following example on v1.12.7:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"
  name: web
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  sessionAffinity: None
  type: LoadBalancer

@mattkeeler
Copy link

Unless I'm missing something, the example above will indeed create an Azure load balancer with TCP and UDP rules for port 80, but Kubernetes itself will only expose the TCP port. I have yet to get UDP traffic working. #75831 appears to be the patch that's needed here.

@feiskyer
Copy link
Member

@mattkeeler You're right, kube-proxy only opens TCP port for service for the above example.

@devteng devteng mentioned this pull request May 31, 2019
Closed
@aledbf aledbf mentioned this pull request Sep 2, 2019
Closed
@MPV
Copy link

MPV commented Jan 7, 2020

KEP progress is being shared in #23880.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feiskyer feiskyer feiskyer left review comments

@justaugustus justaugustus Awaiting requested review from justaugustus

Assignees

@feiskyer feiskyer

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.14
Development

Successfully merging this pull request may close these issues.

Mixed protocols for Azure LoadBalancer is working as expected
9 participants
@andyzhangx @justaugustus @feiskyer @ritazh @k8s-ci-robot @fejta-bot @hargrave81 @mattkeeler @MPV