kubeproxy operator #35

johnsonj · 2019-11-18T21:33:06Z

TODO:

boilerplate
operator can run external to the cluster
make the operator run in the cluster
golden file tests

w/ kubebuilder @ 5f27a7241d: export KUBEBUILDER_ENABLE_PLUGINS=1 kubebuilder init --fetch-deps=false --domain=x-k8s.io --license=apache2 kubebuilder create api --pattern addon --controller=true --example=false --group=addons --kind=KubeProxy --make=false --namespaced=true --resource=true --version=v1alpha

regenerate controller/api code with patch[1] [1] kubernetes-sigs/kubebuilder#1059

This changes implements the basic operator functionality that can bootstrap a new cluster with kubeproxy. The operator currently runs external to the cluster (on your local machine). For an example, see the ./create-cluster.sh script that uses kinder. The change adds 'ClusterCIDR' to the KubeProxy object. It would be interesting to see if we can detect this value when running within the cluster.

timothysc · 2019-12-10T20:23:53Z

cc @neolit123

atoato88 · 2019-12-25T07:53:23Z

kubeproxy/api/v1alpha1/kubeproxy_types.go

+	Status KubeProxyStatus `json:"status,omitempty"`
+}
+
+var _ addonv1alpha1.CommonObject = &KubeProxy{}


Sorry to comment to WIP PR.
Is it better to add var _ addonv1alpha1.Patchable = &KubeProxy{} ?
Because of KubeProxy has PatchSpec interface.
As like here.

fejta-bot · 2020-03-24T08:47:46Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-04-29T14:55:43Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

neolit123 · 2020-04-29T15:10:31Z

/lifecycle frozen

Golden files test for kube proxy

Makes controller run in-cluster

johnsonj · 2020-06-10T14:35:54Z

/remove-lifecycle frozen
/retitle kubeproxy operator

johnsonj · 2020-06-10T15:51:05Z

Thanks to @somtochiama this is 'feature complete'! There's follow up work (eg clean up go.mod, update to latest controller-runtime, integrate with prow tests in this repo) but I think that work should be done in this repo and not this PR.

neolit123 · 2020-06-10T15:55:45Z

hello,

this is 'feature complete'!

is there a roadmap for this operator?
ideally long term we would like to use it a number of SIG Cluster Lifecycle tools including kubeadm, cluster-api.

cc @rosti @fabriziopandini

johnsonj · 2020-06-10T16:03:38Z

@neolit123 - there is no identified roadmap. This is just establishing that things work. There's definitely work to do to make this ready for kubeadm/cluster-api.

We can go through the same exercise as coredns operator: proposal, discussion in the cluster addons meeting minutes, then action (#40)! We do need an owner though. (not sure if @somtochiama is considering this?)

neolit123 · 2020-06-10T17:07:07Z

thanks for the details @johnsonj we just spoke with @rajansandeep and i proposed that he joins the kubeadm office hours so that we can discuss the coredns operator. arguably it is more urgent than the kube-proxy operator, so perhaps we can establish a pattern from it.

johnsonj · 2020-06-10T17:10:26Z

sounds great @neolit123 !

somtochiama · 2020-06-10T17:24:50Z

We can go through the same exercise as coredns operator: proposal, discussion in the cluster addons meeting minutes, then action (#40)! We do need an owner though. (not sure if @somtochiama is considering this?)

I am definitely interested

Tidy up the operator and ensure create-cluster.sh works

johnsonj · 2020-06-11T15:03:52Z

/assign @stealthybox

rajansandeep · 2020-06-11T15:13:32Z

kubeproxy/Dockerfile

+RUN curl -fsSL https://dl.k8s.io/release/v1.17.4/bin/linux/amd64/kubectl > /usr/bin/kubectl
+RUN chmod a+rx /usr/bin/kubectl
+# Build the manager binary
+FROM golang:1.12.5 as builder


We can update to use golang 1.13

rajansandeep · 2020-06-11T15:15:50Z

kubeproxy/go.mod

@@ -0,0 +1,13 @@
+module addon-operators/kubeproxy
+
+go 1.12


We can update to use golang 1.13

rajansandeep · 2020-06-11T15:19:02Z

kubeproxy/create-cluster.sh

@@ -0,0 +1,40 @@
+#!/bin/sh


This seems like a really useful script that can be used to establish e2e/smoketests for all operators.
Should this be moved outside to a test/or hack/ dir?
#63 might give some more context

The script is for creating a kinder cluster without kubeproxy. I could make a PR for one that creates a normal kind or kinder cluster with kube-proxy already installed. Wdyt?

The script is for creating a kinder cluster without kubeproxy.

Yes, this is exactly what's desired. My thinking is once this script creates the cluster, the existing smoketest.go can run the tests on it.

The script is for creating a kinder cluster without kubeproxy.

Yes, this is exactly what's desired. My thinking is once this script creates the cluster, the existing smoketest.go can run the tests on it.

Oooh, I see what you mean. @johnsonj wdyt?

Totally makes sense to me. We'll always need our own kube-up.sh, even in a world where these operators are default we need a way to test with our versions!

Now the question I'd pose is: Copy the shell script, or reimplement it in Go?

I think that the shell script should suffice

rajansandeep

I have a couple of nits that caught my eye.

Updates go versiom

rosti

Thanks @johnsonj !
I guess that the only big concern for me here is the reliance on a deprecated feature (kube-proxy's command line flags) and the lack of component config for kube-proxy itself.

rosti · 2020-06-15T12:08:57Z

kubeproxy/controllers/kubeproxy_controller.go

+	}
+}
+
+func injectFlags(ctx context.Context, object declarative.DeclarativeObject, s string) (string, error) {


Almost all command line flags of kube-proxy are now deprecated (see here). Hence, I would advocate for the use of the kube-proxy component config here.

cc @mtaufen @stealthybox

rosti · 2020-06-15T12:13:23Z

kubeproxy/controllers/kubeproxy_controller.go

+				port = "443"
+			}
+			env := map[string]string{"KUBERNETES_SERVICE_HOST": master, "KUBERNETES_SERVICE_PORT": port}
+			if err := SetEnvironmentVariables(o, env); err != nil {


I don't think that this is the best way for a long term solution. I would recommend having a field in KubeProxySpec that explicitly sets that value. But it's fine to use this approach as long as the value in the CRD is not set.

neolit123 · 2020-06-15T12:49:58Z

kubeproxy/README.md

+## Running in a cluster
+
+1. Create a kinder cluster
+   Ensure kinder is installed. [Installation docs](https://github.com/kubernetes/kubeadm/blob/master/kinder/README.md)


just a FYI, from the kinder docs:

The exported logic in kinder packages can be a subject of change at any point in time, thus using kinder as a library is unsupported.

this message is rather incomplete.

one caveat is that kinder has no support guarantees for usage external to the kubeadm e2e tests even for its CLI. this means that if we have to change something in kinder we would likely do it without a deprecation policy and me might break consumers.

but overall using it for testing the operators seems fine and if one day kubeadm uses the operators we would likely run tests on such a setup using kinder.

justinsb · 2020-07-02T12:36:31Z

Looking at this PR, it sounds like there is generally agreement on the idea, with a desire to make additional changes (moving to componentconfig, splitting out e2e, etc), and it looks like there are several people making those changes sort of concurrently, including PRs to the PR.

As such, I think it's best to merge and then we can all make any additional changes as additional PRs (it is new code, so we're not breaking anyone!)

/approve
/lgtm

k8s-ci-robot · 2020-07-02T12:36:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnsonj, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnsonj,justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

johnsonj added 3 commits September 24, 2019 21:26

kubeproxy: regenerate boilerplate w/ plugin fix

2385c56

regenerate controller/api code with patch[1] [1] kubernetes-sigs/kubebuilder#1059

k8s-ci-robot requested review from stealthybox and timothysc November 18, 2019 21:33

timothysc removed their request for review December 10, 2019 20:23

atoato88 reviewed Dec 25, 2019

View reviewed changes

dholbach mentioned this pull request Feb 27, 2020

GSoC Tracking Issue: Package all the things! #39

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 24, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 29, 2020

k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Apr 29, 2020

somtochiama and others added 6 commits June 2, 2020 05:59

Adds golden file test for Kube proxy

8019c20

Merge pull request #2 from SomtochiAma/golden-file

e5d447b

Golden files test for kube proxy

Makes controller run in-cluster

2eddc47

Documentation for running operator in kinder cluster

3be60ae

Seperates manager.yaml into patches

b965cf3

Merge pull request #1 from SomtochiAma/kubeproxy-complete

4e1497c

Makes controller run in-cluster

k8s-ci-robot removed the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Jun 10, 2020

k8s-ci-robot changed the title ~~WIP: kubeproxy-operator~~ kubeproxy operator Jun 10, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2020

johnsonj force-pushed the kubeproxy branch from 2fcbbec to 49319ea Compare June 10, 2020 14:46

formatting, rename

45a5a45

johnsonj force-pushed the kubeproxy branch from 49319ea to 45a5a45 Compare June 10, 2020 14:47

somtochiama and others added 2 commits June 11, 2020 14:40

Tidy up the operator and ensure create-cluster.sh works

82cac36

Merge pull request #3 from SomtochiAma/kubeproxy-complete

5ccce07

Tidy up the operator and ensure create-cluster.sh works

k8s-ci-robot assigned stealthybox Jun 11, 2020

rajansandeep reviewed Jun 11, 2020

View reviewed changes

kubeproxy/go.mod Outdated

@@ -0,0 +1,13 @@

module addon-operators/kubeproxy

go 1.12

Copy link

Contributor

rajansandeep Jun 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can update to use golang 1.13

rajansandeep reviewed Jun 11, 2020

View reviewed changes

somtochiama and others added 2 commits June 11, 2020 23:24

Updates go versiom

9e0a7df

Merge pull request #4 from SomtochiAma/kubeproxy-complete

7465403

Updates go versiom

rosti reviewed Jun 15, 2020

View reviewed changes

neolit123 reviewed Jun 15, 2020

View reviewed changes

k8s-ci-robot assigned justinsb Jul 2, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2020

k8s-ci-robot merged commit e190793 into kubernetes-sigs:master Jul 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeproxy operator #35

kubeproxy operator #35

johnsonj commented Nov 18, 2019 •

edited

Loading

timothysc commented Dec 10, 2019

atoato88 Dec 25, 2019

fejta-bot commented Mar 24, 2020

fejta-bot commented Apr 29, 2020

neolit123 commented Apr 29, 2020

johnsonj commented Jun 10, 2020

johnsonj commented Jun 10, 2020

neolit123 commented Jun 10, 2020

johnsonj commented Jun 10, 2020

neolit123 commented Jun 10, 2020

johnsonj commented Jun 10, 2020

somtochiama commented Jun 10, 2020

johnsonj commented Jun 11, 2020

rajansandeep Jun 11, 2020

rajansandeep Jun 11, 2020

rajansandeep Jun 11, 2020 •

edited

Loading

somtochiama Jun 11, 2020

rajansandeep Jun 11, 2020

somtochiama Jun 12, 2020

johnsonj Jun 12, 2020

rajansandeep Jun 12, 2020

rajansandeep left a comment

rosti left a comment

rosti Jun 15, 2020

rosti Jun 15, 2020

neolit123 Jun 15, 2020

justinsb commented Jul 2, 2020

k8s-ci-robot commented Jul 2, 2020

kubeproxy operator #35

kubeproxy operator #35

Conversation

johnsonj commented Nov 18, 2019 • edited Loading

timothysc commented Dec 10, 2019

Choose a reason for hiding this comment

fejta-bot commented Mar 24, 2020

fejta-bot commented Apr 29, 2020

neolit123 commented Apr 29, 2020

johnsonj commented Jun 10, 2020

johnsonj commented Jun 10, 2020

neolit123 commented Jun 10, 2020

johnsonj commented Jun 10, 2020

neolit123 commented Jun 10, 2020

johnsonj commented Jun 10, 2020

somtochiama commented Jun 10, 2020

johnsonj commented Jun 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajansandeep Jun 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajansandeep left a comment

Choose a reason for hiding this comment

rosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Jul 2, 2020

k8s-ci-robot commented Jul 2, 2020

johnsonj commented Nov 18, 2019 •

edited

Loading

rajansandeep Jun 11, 2020 •

edited

Loading