-
Notifications
You must be signed in to change notification settings - Fork 39.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
validate host paths on the kubelet for backsteps #47290
validate host paths on the kubelet for backsteps #47290
Conversation
Hi @jhorwit2. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
||
// ValidateHostPath will make sure targetPath: | ||
// 1. does not have any element which is ".." | ||
func ValidateHostPath(targetPath string) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name this more specifically for what it does (ValidateFilePathNoBacksteps or something)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's tons of things this could check that it doesn't (max length, invalid chars, double slashes, etc, and naming it genetically encourages adding those types of checks which could break some callers' assumptions
c584a5d
to
573a9d8
Compare
ac5d0c3
to
ce8cc16
Compare
pkg/kubelet/kubelet_pods.go
Outdated
@@ -61,6 +61,7 @@ import ( | |||
"k8s.io/kubernetes/pkg/util" | |||
"k8s.io/kubernetes/pkg/volume" | |||
"k8s.io/kubernetes/pkg/volume/util/volumehelper" | |||
pvvalidation "k8s.io/kubernetes/pkg/volume/validation" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: volumevalidation, applies to non-PV things as well
path: "/foo/bar", | ||
}, | ||
"invalid hostpath": { | ||
path: "/foo/bar/..", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a test to make sure /foo/..bar is allowed
ce8cc16
to
3103569
Compare
@liggitt should I also add the backstep validation to |
I updated the PR to include apiserver validations also. |
pkg/kubelet/kubelet_pods.go
Outdated
@@ -146,6 +147,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h | |||
|
|||
hostPath = filepath.Join(hostPath, mount.SubPath) | |||
|
|||
err = volumevalidation.ValidatePathNoBacksteps(hostPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have to validate hostPath and mount.SubPath prior to calling filepath.Join, since it cleans/collapses backsteps
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is still outstanding... this needs to be moved above the filepath.Join line above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. Since hostPath is checked in the mounter I moved the check to the top of the conditional and only check the SubPath.
pkg/volume/host_path/host_path.go
Outdated
@@ -103,6 +104,12 @@ func (plugin *hostPathPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volum | |||
if err != nil { | |||
return nil, err | |||
} | |||
|
|||
err = validation.ValidatePathNoBacksteps(hostPathVolumeSource.Path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kubernetes/sig-storage-pr-reviews is the appropriate place to error on an invalid path here or in SetUp?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this question is still outstanding
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Erroring in SetUp(...)
would be better, since that will automatically result in a mount failure and the generation of an event associated with the pod.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@saad-ali It seems weird to me that validation would be in SetUp
and there be no actual setup logic. At that point,SetUp
a misnomer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SetUp is the function that has the mounter actually act on the configuration... the only reason it's a no-op for hostpath is that no setup or checks are currently needed or performed. If you consider a plugin like nfs, if it was given an invalid hostname, it wouldn't know until it tried to perform the mount and failed in SetUp.
pkg/api/validation/validation.go
Outdated
@@ -628,6 +628,9 @@ func validateHostPathVolumeSource(hostPath *api.HostPathVolumeSource, fldPath *f | |||
if len(hostPath.Path) == 0 { | |||
allErrs = append(allErrs, field.Required(fldPath.Child("path"), "")) | |||
} | |||
|
|||
allErrs = append(allErrs, validatePathNoBacksteps(hostPath.Path, fldPath.Child("path"))...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I am in favor of this validation, we need to ensure the following scenarios are handled well:
- if you already have a pod with a path like this, can its status still be updated? Can it be deleted gracefully?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@liggitt Even without the apiserver validation, the kubelet would be performing the same checks on all host volumes. I don't see any easy way around that since, if i'm not mistaken, the kubelet does not have the applied psp for the pod. The kubelet only has the security context to go off of, which does not include any relevant information that would allow the kubelet to check backsteps only for pods with a PSP that has allowed host paths set. Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@derekwaynecarr does the kubelet still use API validation to validate pods? what does it do if it is given a pod that validation determines is invalid? we don't want pods that include backsteps in their mount paths to run, but we don't want to kubelet to fall over if a pod like that existed previously and the kubelet gets upgraded.
// | ||
// This assumes the OS of the apiserver and the nodes are the same. The same check should be done | ||
// on the node to ensure there are no backsteps. | ||
func validatePathNoBacksteps(targetPath string, fldPath *field.Path) field.ErrorList { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function should be uppercase, as being called in other package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it is, there is a similar method in the volume package that is getting called
/unassign |
pkg/api/validation/validation.go
Outdated
@@ -628,6 +628,9 @@ func validateHostPathVolumeSource(hostPath *api.HostPathVolumeSource, fldPath *f | |||
if len(hostPath.Path) == 0 { | |||
allErrs = append(allErrs, field.Required(fldPath.Child("path"), "")) | |||
} | |||
|
|||
allErrs = append(allErrs, validatePathNoBacksteps(hostPath.Path, fldPath.Child("path"))...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neat: put it to else
block to not being executed when path is empty. Or add a return
to the if
body.
pkg/kubelet/kubelet_pods.go
Outdated
@@ -146,6 +147,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h | |||
|
|||
hostPath = filepath.Join(hostPath, mount.SubPath) | |||
|
|||
err = volumevalidation.ValidatePathNoBacksteps(hostPath) | |||
if err != nil { | |||
return nil, fmt.Errorf("unable to provision HostPath `%s`: %v", hostPath, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not use %q
instead of '%s'
?
|
||
plug, err := plugMgr.FindPluginByName("kubernetes.io/host-path") | ||
if err != nil { | ||
t.Errorf("Can't find the plugin by name") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather include plugin name in the error message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it be t.Fatal
to stop execution if we couldn't find a plugin?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TBH, I copy/pasted this part from the other test in this file and didn't pay much attention to it. I agree, should I change both tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, should I change both tests?
I'd change only my own code. Modifying other code makes a diff huge and hard to review.
@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h | |||
return nil, err | |||
} | |||
if mount.SubPath != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jhorwit2 did you ever track down whether we need to be concerned with traversals in mount.MountPath?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't feel like that's a concern. That's the path within the container where the volume is mounted. Escaping out of that directory only yields you directories in the container you can already access; not extra info on the host.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
conceptually, that's what I would expect, but I don't know enough about the implementation to know if that is actually the case. would like an ack on that from @kubernetes/sig-storage-pr-reviews
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This question is also outstanding, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jhorwit2 is right, MountPath
is relative to the inside of the container so it does not matter. mount.SubPath
should be checked.
|
||
// ValidatePathNoBacksteps will make sure the targetPath does not have any element which is ".." | ||
func ValidatePathNoBacksteps(targetPath string) error { | ||
parts := strings.Split(targetPath, string(os.PathSeparator)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
according to https://golang.org/src/os/path_windows.go?s=410:485#L1 windows accepts either /
or \
... not really sure what to do with that.
could do strings.Split(filepath.ToSlash(targetPath), "/")
, I suppose
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there anyone from sig-windows that could comment if this is ideal and/or has any impact on existing values?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc: @bsteciuk to take a look at this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't tested out hostPath volume mounts on Windows yet, though it seems like a reasonable strategy for splitting up the path. I did find a path that looks like it would slip through though.
According to https://en.wikipedia.org/wiki/Path_(computing)#MS-DOS.2FMicrosoft_Windows_style
C:..\File.txt
is a valid path, which points to a file called File.txt located in the parent directory of the current directory on drive C:.
I don't know what the current working directory would be here, but this seems like a case that would have to be checked for.
This small snippet shows it in action, as well as show the first segment which performs the backstepping, though would not be caught by the validation.
package main
import "fmt"
import "os"
import "path/filepath"
import "strings"
func main() {
path := "C:..\\test.file"
os.Create(path)
fmt.Println(strings.Split(filepath.ToSlash(path),"/")[0])
}
The output to stdout is:
C:..
If the binary is located in C:\Users\bsteciuk\go\src\test\main.exe
and I run it from C:\Users\bsteciuk\go\
, then it will create the file C:\Users\bsteciuk\test.file
Edit: Now that I think about it, this validation was added to fix a concern with allowedHostPaths, which would itself likely exclude the path listed above, so this may not actually be a concern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch, but you are correct, allowedHostPaths bounds the hostPath, which the subpath is joined to. I think the split and segment check covers the subpath use case.
pkg/api/validation/validation.go
Outdated
// This assumes the OS of the apiserver and the nodes are the same. The same check should be done | ||
// on the node to ensure there are no backsteps. | ||
func validatePathNoBacksteps(targetPath string, fldPath *field.Path) field.ErrorList { | ||
allErrs := field.ErrorList{} | ||
parts := strings.Split(targetPath, string(os.PathSeparator)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same issue here, maybe strings.Split(filepath.ToSlash(targetPath), "/")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jhorwit2 this change is needed to properly split on windows
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@liggitt pushed
@k8s-bot ok to test |
@jhorwit2 can you rebase/squash, and run |
0bbbebb
to
5c4d499
Compare
other than the windows issue with filepath.ToSlash(), this LGTM. I'd like a second someone from @kubernetes/sig-storage-pr-reviews. A follow-up with @kubernetes/sig-windows-misc on the separator question is fine, as this is strictly better than what we have today |
5c4d499
to
c0b84ad
Compare
Minus the outstanding questions, I've addressed all the concerns 👍 . |
pkg/kubelet/kubelet_pods.go
Outdated
@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h | |||
return nil, err | |||
} | |||
if mount.SubPath != "" { | |||
err = volumevalidation.ValidatePathNoBacksteps(mount.SubPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since IsAbs()
is also platform-dependant, should verify !filepath.IsAbs(mount.SubPath)
before calling ValidatePathNoBacksteps (it's verified in the API, but the OS could differ)
pkg/volume/host_path/host_path.go
Outdated
@@ -103,6 +104,12 @@ func (plugin *hostPathPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volum | |||
if err != nil { | |||
return nil, err | |||
} | |||
|
|||
err = validation.ValidatePathNoBacksteps(hostPathVolumeSource.Path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Erroring in SetUp(...)
would be better, since that will automatically result in a mount failure and the generation of an event associated with the pod.
@@ -138,6 +139,11 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h | |||
return nil, err | |||
} | |||
if mount.SubPath != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jhorwit2 is right, MountPath
is relative to the inside of the container so it does not matter. mount.SubPath
should be checked.
c0b84ad
to
98d1243
Compare
98d1243
to
48b3fb8
Compare
/lgtm |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jhorwit2, saad-ali Assign the PR to them by writing Associated issue: 47107 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
Automatic merge from submit-queue (batch tested with PRs 47626, 47674, 47683, 47290, 47688) |
What this PR does / why we need it:
This PR adds validation on the kubelet to ensure the host path does not contain backsteps that could allow the volume to escape the PSP's allowed host paths. Currently, there is validation done at in API server; however, that does not account for mismatch of OS's on the kubelet vs api server.
Which issue this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close that issue when PR gets merged): fixes #47107Special notes for your reviewer:
cc @liggitt
Release note: