Move unschedulable pods to the active queue if they are not retried for more than 1 minute #72558

denkensk · 2019-01-04T06:15:52Z

What type of PR is this?

/kind bug
/sig scheduling

What this PR does / why we need it:
The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are added to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time.

Which issue(s) this PR fixes:
Fixes #72122

Special notes for your reviewer:

add goroutine to move unschedulable pods to activeq if they are not retried for more than 1 minute

k8s-ci-robot · 2019-01-04T06:15:59Z

Hi @denkensk. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

resouer · 2019-01-04T06:31:49Z

/ok-to-test

resouer · 2019-01-04T19:09:22Z

/assign

I will try to do a round of review.

resouer · 2019-01-04T19:13:10Z

/kind bug
/priority important-longterm

bsalamat

/lgtm
/approve

bsalamat · 2019-01-16T19:10:40Z

pkg/scheduler/internal/queue/scheduling_queue_test.go

+			Namespace: "ns1",
+			UID:       types.UID("tp-mid"),
+		},
+		Spec: v1.PodSpec{


Priority and NominatedNodeName are not really necessary here.

k8s-ci-robot · 2019-01-16T19:11:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, denkensk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/scheduler/OWNERS~~ [bsalamat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Huang-Wei · 2019-01-18T06:30:00Z

@denkensk if this change needs to be back ported into other releases, I guess PR #73022 should also be included?

denkensk · 2019-01-18T06:47:39Z

@denkensk if this change needs to be back ported into other releases, I guess PR #73022 should also be included?

@Huang-Wei Yes, PR #73022 should also be included. I'm not sure if this change needs to be back ported into other releases.

Huang-Wei · 2019-01-18T07:01:24Z

Thanks.

I'm not sure if this change needs to be back ported into other releases.

It seems so, see #72925 (comment). But let's hold for now.

bsalamat · 2019-01-25T18:34:30Z

@denkensk if this change needs to be back ported into other releases, I guess PR #73022 should also be included?

@denkensk @Huang-Wei Yes, this PR should be backported to 1.11+. Could you please send cherrypick PRs?

denkensk · 2019-01-25T19:19:30Z

@denkensk @Huang-Wei Yes, this PR should be backported to 1.11+. Could you please send cherrypick PRs?

Yes, I will send cherrypick PRs.

…58-upstream-release-1.13 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.13

…58-upstream-release-1.11 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.11

…58-upstream-release-1.12 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.12

k8s-ci-robot requested review from Huang-Wei and bsalamat January 4, 2019 06:17

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 4, 2019

denkensk force-pushed the add-goroutine-move-unschedulablepods-to-activeq branch 2 times, most recently from ab21df1 to 1714a9f Compare January 4, 2019 09:07

denkensk changed the title ~~[WIP] add goroutine to move unschedulablepods to activeq regularly~~ add goroutine to move unschedulablepods to activeq regularly Jan 4, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 4, 2019

denkensk changed the title ~~add goroutine to move unschedulablepods to activeq regularly~~ [WIP]add goroutine to move unschedulablepods to activeq regularly Jan 4, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 4, 2019

denkensk changed the title ~~[WIP]add goroutine to move unschedulablepods to activeq regularly~~ add goroutine to move unschedulablepods to activeq regularly Jan 4, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 4, 2019

denkensk changed the title ~~add goroutine to move unschedulablepods to activeq regularly~~ Move unschedulable pods to the active queue if they are not retried for more than 1 minute Jan 4, 2019

k8s-ci-robot assigned resouer Jan 4, 2019

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jan 4, 2019

denkensk force-pushed the add-goroutine-move-unschedulablepods-to-activeq branch 3 times, most recently from 95c0e4a to 40a9a5c Compare January 16, 2019 03:52

add goroutine to move unschedulablepods to activeq regularly

de8cfdc

denkensk force-pushed the add-goroutine-move-unschedulablepods-to-activeq branch from 40a9a5c to de8cfdc Compare January 16, 2019 04:08

bsalamat reviewed Jan 16, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 16, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2019

k8s-ci-robot merged commit d857790 into kubernetes:master Jan 17, 2019

Huang-Wei mentioned this pull request Jan 18, 2019

Scheduler leaves pod stuck in pending after worker node reload #72925

Closed

Huang-Wei mentioned this pull request Jan 19, 2019

scheduler: makes pod less racing so as to be put back into activeQ properly #73078

Merged

bsalamat mentioned this pull request Jan 25, 2019

Pods sharing a PVC on a single node cluster fail to schedule #73216

Closed

cofyc mentioned this pull request Jan 26, 2019

Update PV/PVC assume cache before moving pods to active queue #73305

Closed

k8s-ci-robot added a commit that referenced this pull request Jan 30, 2019

Merge pull request #73454 from denkensk/automated-cherry-pick-of-#725…

0aa225f

…58-upstream-release-1.13 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.13

k8s-ci-robot added a commit that referenced this pull request Feb 2, 2019

Merge pull request #73465 from denkensk/automated-cherry-pick-of-#725…

7ee1441

…58-upstream-release-1.11 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.11

denkensk mentioned this pull request Feb 4, 2019

Don't update the Pod object after each scheduling attempt #73700

Merged

k8s-ci-robot added a commit that referenced this pull request Feb 7, 2019

Merge pull request #73463 from denkensk/automated-cherry-pick-of-#725…

9e8bc14

…58-upstream-release-1.12 Automated cherry pick of #72558: add goroutine to move unschedulablepods to activeq regularly 1.12

denkensk deleted the add-goroutine-move-unschedulablepods-to-activeq branch February 15, 2019 09:32

hiwyw mentioned this pull request Aug 8, 2019

[random] coredns in pending status when creating cluster zdnscloud/singlecloud#177

Closed

cofyc mentioned this pull request Jul 13, 2020

The unit test TestVolumeBinding is flaky #93009

Closed

cofyc mentioned this pull request Jul 30, 2020

Volume Binding PV/PVC Assume Cache should be updated before moving pods to active queue #93552

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Move unschedulable pods to the active queue if they are not retried for more than 1 minute #72558

Move unschedulable pods to the active queue if they are not retried for more than 1 minute #72558

denkensk commented Jan 4, 2019

Uh oh!

k8s-ci-robot commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

bsalamat left a comment

Uh oh!

bsalamat Jan 16, 2019

Uh oh!

k8s-ci-robot commented Jan 16, 2019

Uh oh!

Huang-Wei commented Jan 18, 2019

Uh oh!

denkensk commented Jan 18, 2019 •

edited

Loading

Uh oh!

Huang-Wei commented Jan 18, 2019

Uh oh!

bsalamat commented Jan 25, 2019

Uh oh!

denkensk commented Jan 25, 2019 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Move unschedulable pods to the active queue if they are not retried for more than 1 minute #72558

Move unschedulable pods to the active queue if they are not retried for more than 1 minute #72558

Conversation

denkensk commented Jan 4, 2019

Uh oh!

k8s-ci-robot commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

resouer commented Jan 4, 2019

Uh oh!

bsalamat left a comment

Choose a reason for hiding this comment

Uh oh!

bsalamat Jan 16, 2019

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jan 16, 2019

Uh oh!

Huang-Wei commented Jan 18, 2019

Uh oh!

denkensk commented Jan 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Huang-Wei commented Jan 18, 2019

Uh oh!

bsalamat commented Jan 25, 2019

Uh oh!

denkensk commented Jan 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

denkensk commented Jan 18, 2019 •

edited

Loading

denkensk commented Jan 25, 2019 •

edited

Loading