kubeadm: don't customize etcd selinux label #49328

euank · 2017-07-20T20:40:24Z

The original change that added the unconfined label included a comment
indicating it won't be needed in the future.
See: #33555 (comment)

That time is now. #33663
has landed and means we no longer have to go out of our way to make that
work.

Removing the label also increases security since there wasn't really a
good reason for etcd to be run with such broad selinux privileges.

This also will allow kubeadm to avoid errors on distros without an spc_t
type, such as Gentoo and Container Linux (at the time of writing at
least).

Fixes kubernetes/kubeadm#269

Release note:

kubeadm: Don't set a specific `spc_t` SELinux label on the etcd Static Pod as that is more privs than etcd needs and due to that `spc_t` isn't compatible with some OSes.

The original change that added the unconfined label included a comment indicating it won't be needed in the future. See: kubernetes#33555 (comment) That time is now. kubernetes#33663 has landed and means we no longer have to go out of our way to make that work. Removing the label also increases security since there wasn't really a good reason for etcd to be run with such broad selinux privileges. This also will allow kubeadm to avoid errors on distros without an spc_t type, such as Gentoo and Container Linux (at the time of writing at least). Fixes kubernetes/kubeadm#269

luxas · 2017-07-21T09:08:24Z

/lgtm
/approve

Thanks @euank for fixing this!!
(One thing I can remove from my TODO list 😉)

k8s-github-robot · 2017-07-21T09:08:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: euank, luxas

Associated issue: 269

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kubeadm/OWNERS~~ [luxas]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-07-21T10:00:14Z

Automatic merge from submit-queue (batch tested with PRs 49328, 49285, 49307, 49127, 49163)

wojtek-t · 2017-07-24T11:54:50Z

@luxas - can you please add some release note if you want this cherrypicked?

luxas · 2017-07-24T12:39:22Z

@wojtek-t Done. I'm gonna make a cherrypick for this today.

…49328-upstream-release-1.7 Automatic merge from submit-queue Automated cherry pick of #49498 #49328 Cherry pick of #49498 #49328 on release-1.7. #49498: kubeadm: Make sure --config can be mixed with --skip-* flags #49328: kubeadm: don't customize etcd selinux label

k8s-cherrypick-bot · 2017-07-31T17:08:12Z

Commit found in the "release-1.7" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

jasonbrooks · 2017-09-28T21:30:08Z

@euank How are you installing kubeadm? This change breaks kubeadm on Fedora for me.

luxas · 2017-09-28T21:44:19Z

spc_t doesn't work everywhere, and this makes it possible to run kubeadm on CoreOS at all. As there is no one owning SELinux support for kubeadm, we require `setenforce 0` for SELinux enabled systems If you want to own the area, please get in touch with us in sig cluster lifecycle :)

…

On 29 Sep 2017, at 00:31, Jason Brooks ***@***.***> wrote: @euank How are you installing kubeadm? This change breaks kubeadm on Fedora for me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

euank · 2017-09-28T21:45:07Z

@jasonbrooks Can you describe how it broke specifically (preferably including the docker inspect of the failing etcd container and the audit log for any failures)?

jasonbrooks · 2017-09-28T22:00:03Z

@luxas Fair enough, setenforce 0 is better than nothing.

@euank Does it work in enforcing on coreos? I ask how you install it because the kubeadm docs point to rpm or deb, and neither of those is an option on coreos. I'm wondering if there's something about the presumably containerized install mechanism that's working where the package-based install is not.

----
time->Thu Sep 28 17:56:43 2017
type=PROCTITLE msg=audit(1506635803.032:1011): proctitle=65746364002D2D6164766572746973652D636C69656E742D75726C733D687474703A2F2F3132372E302E302E313A32333739002D2D646174612D6469723D2F7661722F6C69622F65746364002D2D6C697374656E2D636C69656E742D75726C733D687474703A2F2F3132372E302E302E313A32333739
type=SYSCALL msg=audit(1506635803.032:1011): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c820204da0 a2=80241 a3=180 items=0 ppid=11333 pid=11350 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="etcd" exe="/usr/local/bin/etcd" subj=system_u:system_r:container_t:s0:c131,c546 key=(null)
type=AVC msg=audit(1506635803.032:1011): avc:  denied  { create } for  pid=11350 comm="etcd" name=".touch" scontext=system_u:system_r:container_t:s0:c131,c546 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

[
    {
        "Id": "6c48f72c3199570adb1ca2704ad89b1643ec8fac2fada8225f5dd3ab6595980a",
        "Created": "2017-09-28T21:56:58.971082899Z",
        "Path": "etcd",
        "Args": [
            "--advertise-client-urls=http://127.0.0.1:2379",
            "--data-dir=/var/lib/etcd",
            "--listen-client-urls=http://127.0.0.1:2379"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2017-09-28T21:56:59.613461899Z",
            "FinishedAt": "2017-09-28T21:56:59.758665945Z"
        },
        "Image": "sha256:243830dae7dd6ff78859fa1d66098a89e2951a9e95af4ef4d4d2c03d97975771",
        "ResolvConfPath": "/var/lib/docker/containers/24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28/hostname",
        "HostsPath": "/var/lib/docker/containers/24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28/hosts",
        "LogPath": "",
        "Name": "/k8s_etcd_etcd-dhcp-10-171-203.osas.lab.eng.rdu2.redhat.com_kube-system_7d33c3992317f95ec9a171fdf4bd05d6_2",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "system_u:object_r:container_file_t:s0:c131,c546",
        "ProcessLabel": "system_u:system_r:container_t:s0:c131,c546",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/ssl/certs:/etc/ssl/certs",
                "/var/lib/etcd:/var/lib/etcd",
                "/etc/kubernetes:/etc/kubernetes:ro",
                "/var/lib/kubelet/pods/7d33c3992317f95ec9a171fdf4bd05d6/containers/etcd/237f6beb:/dev/termination-log:Z"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "container:24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "container:24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 1000,
            "PidMode": "container:24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined",
                "label=user:system_u",
                "label=role:system_r",
                "label=type:container_t",
                "label=level:s0:c131,c546"
            ],
            "UTSMode": "host",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "kubepods-besteffort-pod7d33c3992317f95ec9a171fdf4bd05d6.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/5687fbd710f323ad4c9dad1e3fc63900765ac8b182a5a7572da2af529f5c6593-init/diff:/var/lib/docker/overlay2/3d37d736f0b38a576c55c0d6433c1366f36aa83fcf30014c4c7a06ea15124b0b/diff:/var/lib/docker/overlay2/a8ca24aa828fa349505bb361bee5d60dd6b367b7eb9c58322d7a79d28c1c103e/diff:/var/lib/docker/overlay2/250d76b935004a194f0df245fecd07db7a875f2aeeede7611472692404b8fa3e/diff",
                "MergedDir": "/var/lib/docker/overlay2/5687fbd710f323ad4c9dad1e3fc63900765ac8b182a5a7572da2af529f5c6593/merged",
                "UpperDir": "/var/lib/docker/overlay2/5687fbd710f323ad4c9dad1e3fc63900765ac8b182a5a7572da2af529f5c6593/diff",
                "WorkDir": "/var/lib/docker/overlay2/5687fbd710f323ad4c9dad1e3fc63900765ac8b182a5a7572da2af529f5c6593/work"
            }
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/etc/ssl/certs",
                "Destination": "/etc/ssl/certs",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/etcd",
                "Destination": "/var/lib/etcd",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/kubernetes",
                "Destination": "/etc/kubernetes",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/7d33c3992317f95ec9a171fdf4bd05d6/containers/etcd/237f6beb",
                "Destination": "/dev/termination-log",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "dhcp-10-171-203.osas.lab.eng.rdu2.redhat.com",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "2379/tcp": {},
                "2380/tcp": {},
                "4001/tcp": {},
                "7001/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "gcr.io/google_containers/etcd-amd64@sha256:d83d3545e06fb035db8512e33bd44afb55dea007a3abd7b17742d3ac6d235940",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "etcd",
                "--advertise-client-urls=http://127.0.0.1:2379",
                "--data-dir=/var/lib/etcd",
                "--listen-client-urls=http://127.0.0.1:2379"
            ],
            "OnBuild": null,
            "Labels": {
                "annotation.io.kubernetes.container.hash": "c7ce287e",
                "annotation.io.kubernetes.container.restartCount": "2",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/7d33c3992317f95ec9a171fdf4bd05d6/etcd_2.log",
                "io.kubernetes.container.name": "etcd",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "etcd-dhcp-10-171-203.osas.lab.eng.rdu2.redhat.com",
                "io.kubernetes.pod.namespace": "kube-system",
                "io.kubernetes.pod.uid": "7d33c3992317f95ec9a171fdf4bd05d6",
                "io.kubernetes.sandbox.id": "24625df1afa31c0c152db1d3a4ef92107e7164d48bd8e63f56501946045dff28"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]

euank · 2017-09-28T22:34:33Z

@jasonbrooks

Container Linux defaults to selinux permissive. I don't personally use kubeadm on it now, so I can't answer the other questions related to it, though presumably with selinux enforcing it would hit the same thing here.

I was expecting to see :Z in the bindmount modes, but it turns out I misunderstood the full scope of the selinux refactor.

It turns out :Z is only auto-applied to configmaps, termination path, and a small handful of other things.
Since it's not applied to hostpath, I guess a label should be added back, though it should be able to be a label specific to just that pod, not a super-privileged one.

Apologies for the regression; I missed the detail that :Z isn't auto-applied here

jasonbrooks · 2017-09-28T23:38:38Z

The spc basically just means unconfined, with the idea being that it's better to unconfine particular containers than to unconfine the entire host, which is what you get with setenforce 0. There's some more info on it at https://danwalsh.livejournal.com/74754.html. But, if kubeadm won't work at all on an selinux-enabled host w/o spc_t in its policy, then that's no good.

luxas · 2017-09-29T21:24:20Z

This was merged to make it possible to use kubeadm on CoreOS at all. Anything else made it fail.
kubeadm can be used on platforms without debs/rpms as well, those are just examples how to package it.
@jasonbrooks Please move further discussion to kubernetes/kubeadm#279

euank · 2017-09-29T21:45:46Z

I do apologize again for misunderstanding the full impact of this change.
The easiest fix might be to create an selinux mapping based on distro detection.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 20, 2017

euank mentioned this pull request Jul 20, 2017

etcd pause container fails to start due to SELinux error on CoreOS kubernetes/kubeadm#269

Closed

k8s-github-robot assigned timothysc and lukemarsden Jul 20, 2017

k8s-github-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. labels Jul 20, 2017

k8s-ci-robot assigned luxas Jul 21, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 21, 2017

luxas added this to the v1.7 milestone Jul 21, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 21, 2017

luxas added cherrypick-candidate and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 21, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 21, 2017

k8s-github-robot merged commit ec9275d into kubernetes:master Jul 21, 2017

euank deleted the etcd-selinux-default branch July 21, 2017 18:03

luxas added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jul 24, 2017

luxas mentioned this pull request Jul 24, 2017

Automated cherry pick of #49498 #49328 #49499

Merged

wojtek-t added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jul 25, 2017

k8s-cherrypick-bot removed the cherrypick-candidate label Jul 31, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: don't customize etcd selinux label #49328

kubeadm: don't customize etcd selinux label #49328

euank commented Jul 20, 2017 •

edited by luxas

Loading

luxas commented Jul 21, 2017

k8s-github-robot commented Jul 21, 2017

k8s-github-robot commented Jul 21, 2017

wojtek-t commented Jul 24, 2017

luxas commented Jul 24, 2017

k8s-cherrypick-bot commented Jul 31, 2017

jasonbrooks commented Sep 28, 2017

luxas commented Sep 28, 2017 via email

euank commented Sep 28, 2017 •

edited

Loading

jasonbrooks commented Sep 28, 2017

euank commented Sep 28, 2017 •

edited

Loading

jasonbrooks commented Sep 28, 2017

luxas commented Sep 29, 2017

euank commented Sep 29, 2017

kubeadm: don't customize etcd selinux label #49328

kubeadm: don't customize etcd selinux label #49328

Conversation

euank commented Jul 20, 2017 • edited by luxas Loading

luxas commented Jul 21, 2017

k8s-github-robot commented Jul 21, 2017

k8s-github-robot commented Jul 21, 2017

wojtek-t commented Jul 24, 2017

luxas commented Jul 24, 2017

k8s-cherrypick-bot commented Jul 31, 2017

jasonbrooks commented Sep 28, 2017

luxas commented Sep 28, 2017 via email

euank commented Sep 28, 2017 • edited Loading

jasonbrooks commented Sep 28, 2017

euank commented Sep 28, 2017 • edited Loading

jasonbrooks commented Sep 28, 2017

luxas commented Sep 29, 2017

euank commented Sep 29, 2017

euank commented Jul 20, 2017 •

edited by luxas

Loading

euank commented Sep 28, 2017 •

edited

Loading

euank commented Sep 28, 2017 •

edited

Loading