Add go profile instrumentation to kubectl #68681

dlespiau · 2018-09-14T16:10:45Z

This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=block -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

What this PR does / why we need it:

This instrumentation is useful to debug abnormal kubectl behaviour or do performance work.

Fixes: #68679

kubectl has gained new --profile and --profile-output options to output go profiles

As an example here's a block profile for the kubectl get nodes command above, rebuilding its cache. The api server isn't local.

bboreham · 2018-09-14T16:21:46Z

/ok-to-test

neolit123 · 2018-09-15T03:10:10Z

/kind feature
up to the maintainers.

dlespiau · 2018-09-17T13:58:49Z

/retest

The pull-kubernetes-e2e-kops-aws failure seems like a flake.
The pull-kubernetes-verify test is still using the previous version of the PR:

I0914 16:31:40.316] Args: --job=pull-kubernetes-verify --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/logs --job=pull-kubernetes-verify --repo=k8s.io/kubernetes=master:94e59f1636a8f8b1566ef35180519d4de5586b79,68681:8653e09382d6becd8a72d0bc06a47e7020008a2b --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/pr-logs --timeout=75 --scenario=kubernetes_verify -- '--branch=${PULL_BASE_REF}' --script=./hack/jenkins/verify-dockerized.sh

8653e09382d6becd8a72d0bc06a47e7020008a2b is the previous commit. Hoping /retest will do the right thing.

Edit: The bot edited the previous comment removing the pull-kubernetes-verify failure!
Edit2: Ah, now I get it, the ci bot only keep one comment with the latest test results, oops!
Edit3: All good!

apelisse

Looks good to me, it should either fail or default if no profile is specified though

apelisse · 2018-09-17T16:48:17Z

pkg/kubectl/cmd/profiling.go

What's the goal of this profile? Does it do something?

This is just a map lookup to check if profileName is the name of an existing profile, validating the input parameter.

Given it prompted a question, it's definitely worth a comment, added it!

pkg/kubectl/cmd/profiling.go

dlespiau · 2018-09-18T10:02:16Z

If no profile is specified, profileName is "" and those functions do nothing, which is the intended behaviour. If profileName is non zero, the profile name is validated by initProfiling to make sure it's the name of a valid profile and fails if it isn't. I think we're covered then.

dims · 2018-09-18T12:54:17Z

/uncc

apelisse · 2018-09-18T15:56:33Z

If no profile is specified, profileName is "" and those functions do nothing

Yeah, I don't think I agree. Running kubectl --profile whatever shouldn't skip profiling, that's too confusing.

dlespiau · 2018-09-18T16:30:24Z

Ah I see what you mean, I think that's a general problem with options though :)

$ kubectl --kubeconfig get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone has the same behaviour:

$ ./_output/bin/kubectl --profile get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone isn't meaningful, just like --kubeconfig alone isn't. The parser takes the next token as the --profile option and tries to find the nodes sub-command, which doesn't exist. I tried to play with -- to terminate the option list and force an empty --profile option but to no avail. --profile on its own is simply not valid, just like --kubeconfig on its own.

To achieve meaningful --profile option, we'd need to have it a boolean option I believe and the UX would become:

$ kubectl --profile --profile-kind=block --profile-output=foo.pprof get nodes

with --profile-kind defaulting to, say, cpu. I have a preference for only having two options.

Just in case the whatever in your example meant a bogus profile name, the current code errors out:

$ ./_output/bin/kubectl --profile whatever get nodes
[...]
unknown profile 'whatever'

apelisse · 2018-09-18T17:04:53Z

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

apelisse · 2018-09-18T17:17:48Z

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

or maybe it's not, maybe you should have an option named "none", and make it the default?

This commit adds two new global options to kubectl: --profile and --profile-output, writing out go profiles to disk to debug interesting and unexpected kubectl behaviour. As an example, here is how to capture a block file, eg. for how long are we blocked on I/O and where? $ kubectl get nodes --profile=mutex -v6 $ go tool pprof -png ./profile.pprof > out.png $ google-chrome out.png Fixes: kubernetes#68679

dlespiau · 2018-09-18T17:31:22Z

Oh I like the "none" idea! Thanks for taking the time to think about it and make a great suggestion!

Now, giving a "" string errors out as such:

./_output/bin/kubectl --profile= get nodes
[...]
unknown profile ''

Updated the code and help string with `"none"``.

apelisse · 2018-09-18T18:23:43Z

/lgtm

Thanks!

k8s-ci-robot · 2018-09-18T18:23:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, dlespiau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/kubectl/OWNERS~~ [apelisse]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dlespiau · 2018-09-19T09:27:38Z

Thanks a lot for the review!

seans3 · 2018-09-20T06:13:48Z

Would it be possible to add a unit test (e.g. profile_test.go)?

dlespiau · 2018-09-21T11:27:40Z

Thinking a bit about it, we could maybe add tests along the lines of:

Check that giving a bogus profile name leads initProfiling to return an error
Check that giving a valid profile name generates a file (and maybe even check that is indeed a pprof file). Maybe cycle through all profile names to check all well-known names generate profiles.

k8s-ci-robot requested review from deads2k and dims September 14, 2018 16:10

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 14, 2018

dlespiau force-pushed the instrument-kubectl branch from de65cbf to 8653e09 Compare September 14, 2018 16:30

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 15, 2018

dlespiau force-pushed the instrument-kubectl branch from 8653e09 to 318da5a Compare September 17, 2018 13:19

apelisse reviewed Sep 17, 2018

View reviewed changes

dlespiau force-pushed the instrument-kubectl branch from 318da5a to 9b31064 Compare September 18, 2018 09:58

k8s-ci-robot removed the request for review from dims September 18, 2018 12:54

dlespiau force-pushed the instrument-kubectl branch from 9b31064 to 5d634e7 Compare September 18, 2018 17:30

k8s-ci-robot assigned apelisse Sep 18, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 18, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 18, 2018

k8s-ci-robot merged commit cc04abc into kubernetes:master Sep 26, 2018

Add go profile instrumentation to kubectl #68681

Add go profile instrumentation to kubectl #68681

Conversation

dlespiau commented Sep 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboreham commented Sep 14, 2018

Uh oh!

neolit123 commented Sep 15, 2018

Uh oh!

dlespiau commented Sep 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apelisse left a comment

Choose a reason for hiding this comment

Uh oh!

apelisse Sep 17, 2018

Choose a reason for hiding this comment

Uh oh!

dlespiau Sep 18, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dlespiau commented Sep 18, 2018

Uh oh!

dims commented Sep 18, 2018

Uh oh!

apelisse commented Sep 18, 2018

Uh oh!

dlespiau commented Sep 18, 2018

Uh oh!

apelisse commented Sep 18, 2018

Uh oh!

apelisse commented Sep 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlespiau commented Sep 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apelisse commented Sep 18, 2018

Uh oh!

k8s-ci-robot commented Sep 18, 2018

Uh oh!

dlespiau commented Sep 19, 2018

Uh oh!

seans3 commented Sep 20, 2018

Uh oh!

dlespiau commented Sep 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dlespiau commented Sep 14, 2018 •

edited

Loading

dlespiau commented Sep 17, 2018 •

edited

Loading

apelisse commented Sep 18, 2018 •

edited

Loading

dlespiau commented Sep 18, 2018 •

edited

Loading