Which jobs are failing?

ci-benchmark-scheduler-perf-master

Which tests are failing?

PreemptionAsync and Unschedulable test cases

Since when has it been failing?

17th Oct 2024

Testgrid link

https://testgrid.k8s.io/sig-scalability-benchmarks#scheduler-perf

Reason for failure (if possible)

PreemptionAsync test is failing because of context deadline error:

    scheduler_perf.go:1427: FATAL ERROR: op 3: error in waiting for pods to get scheduled: at least pod namespace-3/pod-4vfnq is not scheduled: context deadline exceeded
--- FAIL: BenchmarkPerfScheduling/PreemptionAsync/5000Node

Caused by #127829

Unschedulable test is failing because of too high threshold configured:

    scheduler_perf.go:1298: ERROR: op 2: BenchmarkPerfScheduling/Unschedulable/5kNodes/10kPods/namespace-2: expected SchedulingThroughput Average to be higher: got 289.204988, want 400.000000
--- FAIL: BenchmarkPerfScheduling/Unschedulable/5kNodes/10kPods

Caused by #128153
After making these changes, the threshold should be even lower, as it seems a value around 270-280 should be good at the moment.

Anything else we need to know?

You should ignore the first row in testgrid (k8s.io/kubernetes/test/integration/scheduler_perf.scheduler_perf), as it doesn't affect the results and check only the second one (ci-benchmark-scheduler-perf-master.Overall).

Relevant SIG(s)

/sig scheduling