Support running Ubuntu image on GCE node #44744

yguo0905 · 2017-04-21T00:53:33Z

What this PR does / why we need it:

This PR (on top of #44629) contains the script changes for running Ubuntu image on GCE.

Special notes for your reviewer:

We made change in gci/node.yaml and gci/master.yaml to ensure that Kubernetes jobs can start automatically after reboot. This is not needed for GCI but required by Ubuntu. See #44744 (comment) for details. With this change, Ubuntu could use the same provisioning scripts as GCI's.

Ran e2e tests using the following command and all tests passed.

KUBE_GCE_NODE_IMAGE=ubuntu-gke-1604-xenial-v20170420-1 KUBE_GCE_NODE_PROJECT=ubuntu-os-gke-cloud KUBE_NODE_OS_DISTRIBUTION=ubuntu GINKGO_PARALLEL=y GINKGO_PARALLEL_NODES=30 go run hack/e2e.go -- -v --build --up --test --test_args="--ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]" --down

Also tested manually for both GCI and Ubuntu images.

Running Ubuntu image on master is not in the scope and has not been tested.

Release note:
Support Ubuntu 16.04 image on GCE

Using Ubuntu on GCE to run cluster e2e tests requires slightly different node.yaml and master.yaml files than GCI, because Ubuntu uses systemd as PID 1, wheras GCI uses upstart with a systemd delegate. Therefore the e2e tests fail using those files since the kubernetes services are not brought back up after a node/master reboot.

k8s-reviewable · 2017-04-21T00:53:40Z

This change is

yguo0905 · 2017-04-21T00:55:58Z

@chrisglass
@dchen1107

chrisglass

I'm unsure why the ubuntu-specific subdirectory was deleted and instead synlinked to the GCI one, since in my experience this doesn't work.

Specifically, the kubernetes services are not restarted after a system reboot since the systemd target is not enabled, nor does it have an [Install]\nWantedBy= entry.

Or is the assumption that the result of some of the tests should be ignored? (at least one of them uses rebooting nodes to simulate network partition)
Is it assumed that user will never reboot nodes?

chrisglass · 2017-04-21T07:26:50Z

cluster/gce/ubuntu/node.yaml

-      Description=Kubernetes
-
-      [Install]
-      WantedBy=multi-user.target


This is needed by Ubuntu.

The GCI image doesn't have this in its systemd services because GCI uses systemd as a delegate (pid 1 is upstart).

Therefore symlinking ubuntu -> gci does not work: if a node reboots the services won't be brought back up.

That is the reason I originally duplicated some of the files in the subdirectory instead of symlinking, the node.yaml and master.yaml files need to be different.

Or does this imply that GCI will change to use systemd as pid 1?

chrisglass · 2017-04-21T07:27:44Z

cluster/gce/ubuntu/node.yaml

- - systemctl enable kubelet-monitor.service
- - systemctl enable kube-logrotate.timer
- - systemctl enable kube-logrotate.service
- - systemctl enable kubernetes.target


This is needed by Ubuntu.

This extra "enable" call ensures that the appropriate symlink is created by systemd (so that it is brought back up after a reboot).

chrisglass · 2017-04-21T07:29:09Z

cluster/gce/ubuntu/node-helper.sh

-  ensure-gci-metadata-files
-  create-node-template "$template_name" "${scope_flags[*]}" \
-    "kube-env=${KUBE_TEMP}/node-kube-env.yaml" \
-    "user-data=${KUBE_ROOT}/cluster/gce/ubuntu/node.yaml" \


The existence of this file was to make sure the approrpiate node.yaml file is passed to the node instance (an ubuntu specific one, not the GCI one).

chrisglass · 2017-04-21T07:29:29Z

cluster/gce/ubuntu/master.yaml

- - systemctl enable kubelet-monitor.service
- - systemctl enable kube-logrotate.timer
- - systemctl enable kube-logrotate.service
- - systemctl enable kubernetes.target


This is needed by Ubuntu.

This extra "enable" call ensures that the appropriate symlink is created by systemd (so that it is brought back up after a reboot).

chrisglass · 2017-04-21T07:29:56Z

cluster/gce/ubuntu/master.yaml

-      Description=Kubernetes
-
-      [Install]
-      WantedBy=multi-user.target


This is needed by Ubuntu.

The GCI image doesn't have this in its systemd services because GCI uses systemd as a delegate (pid 1 is upstart).

Therefore symlinking ubuntu -> gci does not work: if a node reboots the services won't be brought back up.

That is the reason I originally duplicated some of the files in the subdirectory instead of symlinking, the node.yaml and master.yaml files need to be different.

yguo0905 · 2017-04-21T17:52:41Z

You are right. I wasn't aware of the reboot issue. The tests I ran (the slow, serial, disruptive, flaky and feature tests were skipped) didn't cover this case.
I will keep the original commit and set up Jenkins jobs to run more tests against it.

spxtr · 2017-04-21T17:53:03Z

@k8s-bot test this

chrisglass · 2017-04-24T18:13:48Z

Since this is basically the same as #44629 and I don't have push access to this branch, I suggest this to be closed and efforts focused on #44629 instead.

dchen1107 · 2017-04-24T17:48:18Z

cluster/common.sh

-  if [[ "${master}" == "true" && ("${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "container-linux") ]] || \
-     [[ "${master}" == "false" && ("${NODE_OS_DISTRIBUTION}" == "trusty" || "${NODE_OS_DISTRIBUTION}" == "gci" || "${NODE_OS_DISTRIBUTION}" == "container-linux") ]] ; then
+  if [[ "${master}" == "true" && ("${MASTER_OS_DISTRIBUTION}" == "trusty" || "${MASTER_OS_DISTRIBUTION}" == "gci" || "${MASTER_OS_DISTRIBUTION}" == "container-linux") || "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]] || \
+     [[ "${master}" == "false" && ("${NODE_OS_DISTRIBUTION}" == "trusty" || "${NODE_OS_DISTRIBUTION}" == "gci" || "${NODE_OS_DISTRIBUTION}" == "container-linux") || "${NODE_OS_DISTRIBUTION}" = "ubuntu" ]] ; then


We should remove all references to trusty since the trusty support was removed a while back. Let's do it in a separate pr.

dchen1107 · 2017-04-24T17:51:50Z

cluster/gce/ubuntu/master-helper.sh

@@ -0,0 +1,111 @@
+#!/bin/bash
+
+# Copyright 2016 The Kubernetes Authors.


s/2016/2017?

Changed in f07669b

dchen1107 · 2017-04-24T18:17:07Z

cluster/gce/ubuntu/master-helper.sh

+    --tags "${MASTER_TAG}" \
+    --network "${NETWORK}" \
+    --scopes "storage-ro,compute-rw,monitoring,logging-write" \
+    --can-ip-forward \


@chrisglass By comparing gce/ubuntu/master-helper.sh and gce/gci/master-helper.sh, adding --can-ip-forward upon creating the instance is the only real difference here. Why do we need this if GCI / COS doesn't require it? cc/ @roberthbailey @zmerlynn

The reason I care this subtle difference is that I want to make sure the difference between two package as minimal as possible.

I suspect I "forked" the script when GCI still had --can-ip-forward. I am fine with dropping it.

I'm surprised that COS doesn't require the --can-ip-forward flag.

@bowei

@dchen1107 -- Is this because we never contact a pod IP on the master from any other node?

Both master-helper.sh and node-helper.sh (via create-node-template) call make-gcloud-network-argument, which sets --can-ip-forward unless KUBE_GCE_ENABLE_IP_ALIASES is set. So COS is using this flag and we are good.

dchen1107 · 2017-04-24T18:26:36Z

cluster/gce/ubuntu/master-helper.sh

+
+  echo "${kube_env}" > ${KUBE_TEMP}/master-kube-env.yaml
+  get-metadata "${existing_master_zone}" "${existing_master_name}" cluster-name > "${KUBE_TEMP}/cluster-name.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-update-strategy > "${KUBE_TEMP}/gci-update.txt"


GCI specific variable, no need here.

dchen1107 · 2017-04-24T18:27:16Z

cluster/gce/ubuntu/master-helper.sh

+  echo "${kube_env}" > ${KUBE_TEMP}/master-kube-env.yaml
+  get-metadata "${existing_master_zone}" "${existing_master_name}" cluster-name > "${KUBE_TEMP}/cluster-name.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-update-strategy > "${KUBE_TEMP}/gci-update.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-ensure-gke-docker > "${KUBE_TEMP}/gci-ensure-gke-docker.txt"


GCI specific legacy variable, no need here, and should be removed from gci too.

dchen1107 · 2017-04-24T18:27:26Z

cluster/gce/ubuntu/master-helper.sh

+  get-metadata "${existing_master_zone}" "${existing_master_name}" cluster-name > "${KUBE_TEMP}/cluster-name.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-update-strategy > "${KUBE_TEMP}/gci-update.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-ensure-gke-docker > "${KUBE_TEMP}/gci-ensure-gke-docker.txt"
+  get-metadata "${existing_master_zone}" "${existing_master_name}" gci-docker-version > "${KUBE_TEMP}/gci-docker-version.txt"


dchen1107 · 2017-04-24T18:56:11Z

cluster/gce/ubuntu/master.yaml

+ - systemctl enable kubelet-monitor.service
+ - systemctl enable kube-logrotate.timer
+ - systemctl enable kube-logrotate.service
+ - systemctl enable kubernetes.target


Based on my understanding, this is the only line (another one in node.yaml) must required to creating a separate gce/ubuntu folder besides gce/common.sh & util.sh. I am wondering if we can introduce the same:

systemctl enable kubernetes.target
before start kubernetes.target. If we could, we can hugely simplify the codebase, and avoid future unnecessary devergening and minimize the maintenance cost.

cc/ @kubernetes/sig-cluster-lifecycle-api-reviews

I've looked into this. In GCI, the cloud-init's specs in /var/lib/cloud are not persisted across reboots, so cloud-init will always run everything as if it's on a newly created machine¹. Therefore, systemctl enable kubernetes.target is useless in this case.

However, this is not the case for Ubuntu, where cloud-init will skip those steps whose frequency is once. That's why we need to have kubernetes.target required by state multi-user.target and have it enabled. But I'm not sure if it needs to be strictly started after docker.service though. (BTW, the After should be under section [Unit] rather than [Install]).

Therefore, I think it should be safe to make the changes for GCI and change ubuntu to be a symlink to gci. Both GCI and Ubuntu should be working with this change. (I'm not sure if Ubuntu should have the same behavior as GCI wrt cloud-init though.)

@roberthbailey and @kubernetes/sig-cluster-lifecycle-api-reviews, can you please confirm?

¹ This is because the upstart jobs are in /etc/init, which is not persisted across reboots in GCI. In this case, the cloud-init will be rerun in each reboot but it will not recreate the jobs because their run frequency is once. In order to fix this, GCI needs to reset /var/lib/cloud during restart to make cloud-init always run everything.

dchen1107 · 2017-04-24T18:56:47Z

cluster/gce/ubuntu/node-helper.sh

+    "user-data=${KUBE_ROOT}/cluster/gce/ubuntu/node.yaml" \
+    "configure-sh=${KUBE_ROOT}/cluster/gce/gci/configure.sh" \
+    "cluster-name=${KUBE_TEMP}/cluster-name.txt" \
+    "gci-update-strategy=${KUBE_TEMP}/gci-update.txt" \


no need this.

dchen1107 · 2017-04-24T18:56:56Z

cluster/gce/ubuntu/node-helper.sh

+    "configure-sh=${KUBE_ROOT}/cluster/gce/gci/configure.sh" \
+    "cluster-name=${KUBE_TEMP}/cluster-name.txt" \
+    "gci-update-strategy=${KUBE_TEMP}/gci-update.txt" \
+    "gci-ensure-gke-docker=${KUBE_TEMP}/gci-ensure-gke-docker.txt" \


dchen1107 · 2017-04-24T18:57:03Z

cluster/gce/ubuntu/node-helper.sh

+    "cluster-name=${KUBE_TEMP}/cluster-name.txt" \
+    "gci-update-strategy=${KUBE_TEMP}/gci-update.txt" \
+    "gci-ensure-gke-docker=${KUBE_TEMP}/gci-ensure-gke-docker.txt" \
+    "gci-docker-version=${KUBE_TEMP}/gci-docker-version.txt"


dchen1107 · 2017-04-24T18:57:23Z

cluster/gce/ubuntu/node.yaml

+ - systemctl enable kubelet-monitor.service
+ - systemctl enable kube-logrotate.timer
+ - systemctl enable kube-logrotate.service
+ - systemctl enable kubernetes.target


Same comment as the one in master.yaml

…es the gci's [master|node].yaml to enable kubernetes.target. This enables Ubuntu to use the same provisioning scripts as GCI's. The change for enabling kubernetes.target is needed for Ubuntu but a no-op for GCI.

yguo0905 · 2017-04-25T23:40:00Z

I've changed ubuntu to be a symlink to gci and made changes to start kubernetes.target automatically after reboot. Tested manually for both GCI and Ubuntu images (for reboots). Please take another look.

yguo0905 · 2017-04-26T17:06:50Z

Besides my manual tests, also confirmed with @adityakali, the changes in [master|node].yaml should be a no-op for GCI.

dchen1107 · 2017-04-26T17:24:51Z

I updated the release-notes a little bit: Support Ubuntu 16.04 image on GCE

/lgtm

k8s-github-robot · 2017-04-26T17:25:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, yguo0905

Needs approval from an approver in each of these OWNERS Files:

~~cluster/OWNERS~~ [dchen1107]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-04-26T18:12:56Z

Automatic merge from submit-queue

k8s-cherrypick-bot · 2017-05-02T00:31:22Z

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

Automatic merge from submit-queue Cherry pick #44744 on release-1.6 **What this PR does / why we need it**: Cherry pick of #44744 on release-1.6. The changes need to cherry picked are relatively minimal so the risky is low. This allows us to run Ubuntu e2e tests against 1.6 branch. **Special notes for your reviewer**: **Release note**: ``` Support Ubuntu 16.04 image on GCE ```

k8s-cherrypick-bot · 2017-05-04T00:40:25Z

Commit found in the "release-1.6" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

mikedanese · 2017-05-15T17:13:47Z

cluster/gce/util.sh

@@ -1,6 +1,6 @@
 #!/bin/bash

-# Copyright 2014 The Kubernetes Authors.
+# Copyright 2017 The Kubernetes Authors.


We don't update copyright dates. The dates should be when the file was added.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 21, 2017

k8s-github-robot assigned jszczepkowski and eparis Apr 21, 2017

k8s-github-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. release-note-label-needed labels Apr 21, 2017

chrisglass reviewed Apr 21, 2017

View reviewed changes

yguo0905 force-pushed the ubuntu-gce-cluster-tests branch from 4ee450f to cf23d4c Compare April 21, 2017 17:48

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 21, 2017

dchen1107 assigned dchen1107 and chrisglass and unassigned eparis and jszczepkowski Apr 24, 2017

dchen1107 mentioned this pull request Apr 24, 2017

Add Ubuntu as a platform for e2e tests on GCE #44629

Closed

dchen1107 assigned roberthbailey Apr 24, 2017

dchen1107 requested changes Apr 24, 2017

View reviewed changes

dchen1107 reviewed Apr 24, 2017

View reviewed changes

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 25, 2017

dchen1107 approved these changes Apr 26, 2017

View reviewed changes

dchen1107 added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Apr 26, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 26, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 26, 2017

k8s-github-robot merged commit 6d17ab3 into kubernetes:master Apr 26, 2017

yguo0905 changed the title ~~Support running Ubuntu image on GCE~~ Support running Ubuntu image on GCE node Apr 27, 2017

yguo0905 mentioned this pull request May 2, 2017

Cherry pick #44744 on release-1.6 #45197

Merged

dchen1107 added the cherrypick-candidate label May 2, 2017

k8s-cherrypick-bot removed the cherrypick-candidate label May 2, 2017

dchen1107 added the cherrypick-candidate label May 2, 2017

dchen1107 added this to the v1.6 milestone May 2, 2017

k8s-cherrypick-bot removed the cherrypick-candidate label May 4, 2017

mikedanese reviewed May 15, 2017

View reviewed changes

		@@ -0,0 +1,111 @@
		#!/bin/bash

		# Copyright 2016 The Kubernetes Authors.

Support running Ubuntu image on GCE node #44744

Support running Ubuntu image on GCE node #44744

Conversation

yguo0905 commented Apr 21, 2017 • edited Loading

k8s-reviewable commented Apr 21, 2017

yguo0905 commented Apr 21, 2017

chrisglass left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yguo0905 commented Apr 21, 2017

spxtr commented Apr 21, 2017

chrisglass commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yguo0905 Apr 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dchen1107 Apr 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yguo0905 commented Apr 25, 2017

yguo0905 commented Apr 26, 2017

dchen1107 commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

k8s-cherrypick-bot commented May 2, 2017

k8s-cherrypick-bot commented May 4, 2017

Choose a reason for hiding this comment

yguo0905 commented Apr 21, 2017 •

edited

Loading

chrisglass left a comment •

edited

Loading

yguo0905 Apr 26, 2017 •

edited

Loading

dchen1107 Apr 24, 2017 •

edited

Loading