Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validate host paths on the kubelet for backsteps #47290

Merged

Conversation

jhorwit2
Copy link
Contributor

@jhorwit2 jhorwit2 commented Jun 10, 2017

What this PR does / why we need it:

This PR adds validation on the kubelet to ensure the host path does not contain backsteps that could allow the volume to escape the PSP's allowed host paths. Currently, there is validation done at in API server; however, that does not account for mismatch of OS's on the kubelet vs api server.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #47107

Special notes for your reviewer:

cc @liggitt

Release note:

Paths containing backsteps (for example, "../bar") are no longer allowed in hostPath volume paths, or in volumeMount subpaths

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 10, 2017
@k8s-ci-robot
Copy link
Contributor

Hi @jhorwit2. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 10, 2017
@k8s-github-robot k8s-github-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-label-needed labels Jun 10, 2017

// ValidateHostPath will make sure targetPath:
// 1. does not have any element which is ".."
func ValidateHostPath(targetPath string) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name this more specifically for what it does (ValidateFilePathNoBacksteps or something)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's tons of things this could check that it doesn't (max length, invalid chars, double slashes, etc, and naming it genetically encourages adding those types of checks which could break some callers' assumptions

@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from c584a5d to 573a9d8 Compare June 10, 2017 15:13
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 10, 2017
@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from ac5d0c3 to ce8cc16 Compare June 10, 2017 15:17
@@ -61,6 +61,7 @@ import (
"k8s.io/kubernetes/pkg/util"
"k8s.io/kubernetes/pkg/volume"
"k8s.io/kubernetes/pkg/volume/util/volumehelper"
pvvalidation "k8s.io/kubernetes/pkg/volume/validation"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: volumevalidation, applies to non-PV things as well

path: "/foo/bar",
},
"invalid hostpath": {
path: "/foo/bar/..",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test to make sure /foo/..bar is allowed

@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from ce8cc16 to 3103569 Compare June 10, 2017 15:18
@jhorwit2
Copy link
Contributor Author

@liggitt should I also add the backstep validation to HostPath.Path on the pod? That way we catch hopefully most of these before they hit the node.

@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 10, 2017
@jhorwit2
Copy link
Contributor Author

I updated the PR to include apiserver validations also.

@@ -146,6 +147,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h

hostPath = filepath.Join(hostPath, mount.SubPath)

err = volumevalidation.ValidatePathNoBacksteps(hostPath)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to validate hostPath and mount.SubPath prior to calling filepath.Join, since it cleans/collapses backsteps

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is still outstanding... this needs to be moved above the filepath.Join line above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Since hostPath is checked in the mounter I moved the check to the top of the conditional and only check the SubPath.

@@ -103,6 +104,12 @@ func (plugin *hostPathPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volum
if err != nil {
return nil, err
}

err = validation.ValidatePathNoBacksteps(hostPathVolumeSource.Path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kubernetes/sig-storage-pr-reviews is the appropriate place to error on an invalid path here or in SetUp?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this question is still outstanding

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Erroring in SetUp(...) would be better, since that will automatically result in a mount failure and the generation of an event associated with the pod.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saad-ali It seems weird to me that validation would be in SetUp and there be no actual setup logic. At that point,SetUp a misnomer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SetUp is the function that has the mounter actually act on the configuration... the only reason it's a no-op for hostpath is that no setup or checks are currently needed or performed. If you consider a plugin like nfs, if it was given an invalid hostname, it wouldn't know until it tried to perform the mount and failed in SetUp.

@@ -628,6 +628,9 @@ func validateHostPathVolumeSource(hostPath *api.HostPathVolumeSource, fldPath *f
if len(hostPath.Path) == 0 {
allErrs = append(allErrs, field.Required(fldPath.Child("path"), ""))
}

allErrs = append(allErrs, validatePathNoBacksteps(hostPath.Path, fldPath.Child("path"))...)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I am in favor of this validation, we need to ensure the following scenarios are handled well:

  • if you already have a pod with a path like this, can its status still be updated? Can it be deleted gracefully?

Copy link
Contributor Author

@jhorwit2 jhorwit2 Jun 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt Even without the apiserver validation, the kubelet would be performing the same checks on all host volumes. I don't see any easy way around that since, if i'm not mistaken, the kubelet does not have the applied psp for the pod. The kubelet only has the security context to go off of, which does not include any relevant information that would allow the kubelet to check backsteps only for pods with a PSP that has allowed host paths set. Thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@derekwaynecarr does the kubelet still use API validation to validate pods? what does it do if it is given a pod that validation determines is invalid? we don't want pods that include backsteps in their mount paths to run, but we don't want to kubelet to fall over if a pod like that existed previously and the kubelet gets upgraded.

//
// This assumes the OS of the apiserver and the nodes are the same. The same check should be done
// on the node to ensure there are no backsteps.
func validatePathNoBacksteps(targetPath string, fldPath *field.Path) field.ErrorList {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function should be uppercase, as being called in other package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is, there is a similar method in the volume package that is getting called

@jsafrane
Copy link
Member

/unassign
/assing @liggitt

@@ -628,6 +628,9 @@ func validateHostPathVolumeSource(hostPath *api.HostPathVolumeSource, fldPath *f
if len(hostPath.Path) == 0 {
allErrs = append(allErrs, field.Required(fldPath.Child("path"), ""))
}

allErrs = append(allErrs, validatePathNoBacksteps(hostPath.Path, fldPath.Child("path"))...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat: put it to else block to not being executed when path is empty. Or add a return to the if body.

@@ -146,6 +147,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h

hostPath = filepath.Join(hostPath, mount.SubPath)

err = volumevalidation.ValidatePathNoBacksteps(hostPath)
if err != nil {
return nil, fmt.Errorf("unable to provision HostPath `%s`: %v", hostPath, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use %q instead of '%s'?


plug, err := plugMgr.FindPluginByName("kubernetes.io/host-path")
if err != nil {
t.Errorf("Can't find the plugin by name")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather include plugin name in the error message.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be t.Fatal to stop execution if we couldn't find a plugin?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, I copy/pasted this part from the other test in this file and didn't pay much attention to it. I agree, should I change both tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, should I change both tests?

I'd change only my own code. Modifying other code makes a diff huge and hard to review.

@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h
return nil, err
}
if mount.SubPath != "" {
Copy link
Member

@liggitt liggitt Jun 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhorwit2 did you ever track down whether we need to be concerned with traversals in mount.MountPath?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel like that's a concern. That's the path within the container where the volume is mounted. Escaping out of that directory only yields you directories in the container you can already access; not extra info on the host.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conceptually, that's what I would expect, but I don't know enough about the implementation to know if that is actually the case. would like an ack on that from @kubernetes/sig-storage-pr-reviews

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This question is also outstanding, correct?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhorwit2 is right, MountPath is relative to the inside of the container so it does not matter. mount.SubPath should be checked.


// ValidatePathNoBacksteps will make sure the targetPath does not have any element which is ".."
func ValidatePathNoBacksteps(targetPath string) error {
parts := strings.Split(targetPath, string(os.PathSeparator))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to https://golang.org/src/os/path_windows.go?s=410:485#L1 windows accepts either / or \... not really sure what to do with that.

could do strings.Split(filepath.ToSlash(targetPath), "/"), I suppose

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there anyone from sig-windows that could comment if this is ideal and/or has any impact on existing values?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @bsteciuk to take a look at this

Copy link
Contributor

@bsteciuk bsteciuk Jun 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhorwit2 @michmike

I haven't tested out hostPath volume mounts on Windows yet, though it seems like a reasonable strategy for splitting up the path. I did find a path that looks like it would slip through though.

According to https://en.wikipedia.org/wiki/Path_(computing)#MS-DOS.2FMicrosoft_Windows_style
C:..\File.txt is a valid path, which points to a file called File.txt located in the parent directory of the current directory on drive C:.

I don't know what the current working directory would be here, but this seems like a case that would have to be checked for.

This small snippet shows it in action, as well as show the first segment which performs the backstepping, though would not be caught by the validation.

package main

import "fmt"
import "os"
import "path/filepath" 
import "strings"

func main() {
	path := "C:..\\test.file"
	os.Create(path)
	fmt.Println(strings.Split(filepath.ToSlash(path),"/")[0])
}

The output to stdout is:
C:..

If the binary is located in C:\Users\bsteciuk\go\src\test\main.exe and I run it from C:\Users\bsteciuk\go\, then it will create the file C:\Users\bsteciuk\test.file

Edit: Now that I think about it, this validation was added to fix a concern with allowedHostPaths, which would itself likely exclude the path listed above, so this may not actually be a concern.

Copy link
Member

@liggitt liggitt Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, but you are correct, allowedHostPaths bounds the hostPath, which the subpath is joined to. I think the split and segment check covers the subpath use case.

// This assumes the OS of the apiserver and the nodes are the same. The same check should be done
// on the node to ensure there are no backsteps.
func validatePathNoBacksteps(targetPath string, fldPath *field.Path) field.ErrorList {
allErrs := field.ErrorList{}
parts := strings.Split(targetPath, string(os.PathSeparator))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same issue here, maybe strings.Split(filepath.ToSlash(targetPath), "/")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhorwit2 this change is needed to properly split on windows

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt pushed

@liggitt liggitt added this to the v1.7 milestone Jun 13, 2017
@liggitt liggitt added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed release-note-label-needed labels Jun 13, 2017
@liggitt
Copy link
Member

liggitt commented Jun 13, 2017

@k8s-bot ok to test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 13, 2017
@liggitt
Copy link
Member

liggitt commented Jun 13, 2017

@jhorwit2 can you rebase/squash, and run hack/update-bazel.sh to get the tests running?

@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch 3 times, most recently from 0bbbebb to 5c4d499 Compare June 14, 2017 00:45
@liggitt
Copy link
Member

liggitt commented Jun 14, 2017

other than the windows issue with filepath.ToSlash(), this LGTM. I'd like a second someone from @kubernetes/sig-storage-pr-reviews.

A follow-up with @kubernetes/sig-windows-misc on the separator question is fine, as this is strictly better than what we have today

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/windows Categorizes an issue or PR as relevant to SIG Windows. labels Jun 14, 2017
@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from 5c4d499 to c0b84ad Compare June 16, 2017 03:02
@jhorwit2
Copy link
Contributor Author

Minus the outstanding questions, I've addressed all the concerns 👍 .

@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h
return nil, err
}
if mount.SubPath != "" {
err = volumevalidation.ValidatePathNoBacksteps(mount.SubPath)
Copy link
Member

@liggitt liggitt Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since IsAbs() is also platform-dependant, should verify !filepath.IsAbs(mount.SubPath) before calling ValidatePathNoBacksteps (it's verified in the API, but the OS could differ)

@@ -103,6 +104,12 @@ func (plugin *hostPathPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volum
if err != nil {
return nil, err
}

err = validation.ValidatePathNoBacksteps(hostPathVolumeSource.Path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Erroring in SetUp(...) would be better, since that will automatically result in a mount failure and the generation of an event associated with the pod.

@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h
return nil, err
}
if mount.SubPath != "" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhorwit2 is right, MountPath is relative to the inside of the container so it does not matter. mount.SubPath should be checked.

@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from c0b84ad to 98d1243 Compare June 16, 2017 20:41
@jhorwit2 jhorwit2 force-pushed the jah/hostpath-psp-backstep-check branch from 98d1243 to 48b3fb8 Compare June 16, 2017 20:52
@saad-ali
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 16, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jhorwit2, saad-ali
We suggest the following additional approver: thockin

Assign the PR to them by writing /assign @thockin in a comment when ready.

Associated issue: 47107

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@jhorwit2 jhorwit2 changed the title WIP validate host paths on the kubelet for backsteps validate host paths on the kubelet for backsteps Jun 16, 2017
@saad-ali saad-ali added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 47626, 47674, 47683, 47290, 47688)

@k8s-github-robot k8s-github-robot merged commit 098e1df into kubernetes:master Jun 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/security cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

host path volumes allow backsteps, PSP allowedHostPaths only guard prefix
10 participants