This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

/cc @macsko @pohly @sanposhiho

Adjusting the limits is very cumbersome. Yes, there are graphs of historical runs, but throughputs varies a lot (in all tests). It would be good to normalize results somehow, otherwise their value is a bit limited. Moreover, running tests locally gives very different results than the ones on ci machines, so it's hard to predict right limits. For instance SchedulingBasic on local machine gives 1000+, but CI gives 300+.

I also noticed that the tests does not show logs anymore if there are no failures, so there is no possibility to verify the numbers even when triggered in PR tests.

Do you have any recommendations to adjust limits other than watching historical runs?

Well, I'm not sure if we could normalize the results without running the tests multiple times in a single run (which sounds ...not elegant?).

So, the scheduler-perf alert email arrives when there's three (iirc) consecutive failures of the CI. So, in that sense, we kinda normalized the results for the alert already: the min throughput of three runs is compared against the threshold, and if it's lower, the email would come.
So, we can actually ignore some spikes in the test results. I assume a few reds (due to some spikes) in the testgrid is accepted for this job, unless there's any team hoping to keep all jobs completely green. (and I guess there isn't? because scheduler-perf was all red until very recently either way)

It would be good to normalize results somehow

I was thinking about it and I think it's not possible (in a simple way). The performance drops during the test are not constant, what results in only a few tests being affected by it. A bit of flakiness in the tests is not critical, especially given that alerts are generated for 3 consecutive failures as @sanposhiho pointed out.

Moreover, running tests locally gives very different results than the ones on ci machines, so it's hard to predict right limits.

That's why it's better to set either pessimistic thresholds for new tests or no thresholds. They could be always adjusted after obtaining some history.

I also noticed that the tests does not show logs anymore if there are no failures, so there is no possibility to verify the numbers even when triggered in PR tests.

That's right. For some reason result json is not written into job's artifacts for pull-kubernetes-scheduler-perf presubmit. There were some PRs trying to fix this, but still without a success.

/lgtm

LGTM label has been added.

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dom4ha, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

That's right. For some reason result json is not written into job's artifacts for pull-kubernetes-scheduler-perf presubmit. There were some PRs trying to fix this, but still without a success.

I wasn't trying to fix that. As far as I remember, there's simply a command line parameter missing in the pull job. But I am not sure whether the pull job really runs in the same controlled environment as the CI job - that would have to be checked. Otherwise the results are not comparable and/or more flaky.

Well, I'm not sure if we could normalize the results without running the tests multiple times in a single run (which sounds ...not elegant?).

True. I was thinking about normalizing them post execution, for instance during performance graph creation. I wasn't aware that alerting does that, but using a rolling average of n past runs might be good enough and could let us more easily notice the trend (although with some additional delay). Taking into consideration overall tests duration could give even more stable results.

That's why it's better to set either pessimistic thresholds for new tests or no thresholds. They could be always adjusted after obtaining some history.

Setting pessimistic or no threshold sounds like the right approach for now.

That's right. For some reason result json is not written into job's artifacts for pull-kubernetes-scheduler-perf presubmit. There were some PRs trying to fix this, but still without a success.

Something must have changed recently, as tests triggered in PRs used to have full log, but the recent runs log only test verdicts.

Alerting doesn't average performance results. But it provides a "buffer" against being alerted for every single failed run. So as long as we get some good runs before a failed one, we won't get notified.