Skip to content

Commit 75c1fcb

Browse files
mdjastrzebskimdjastrzebski
authored
chore: stability harness (#588)
1 parent 5c8258f commit 75c1fcb

20 files changed

Lines changed: 368 additions & 72 deletions

‎.changeset/floppy-hands-marry.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
"@callstack/reassure-compare": minor
3+
"reassure-test-app": minor
4+
"reassure": minor
5+
---
6+
7+
feat: calculate stability measures for test runs

‎.yarnrc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
nodeLinker: node-modules
22
npmMinimalAgeGate: '3d'
3+
npmPreapprovedPackages:
4+
- react-test-config

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Reassure is a performance testing toolkit for React and React Native, structured
2222
## Further reading
2323

2424
- [Project structure](docs/agents/structure.md)
25+
- [Domain language](docs/agents/domain.md)
2526
- [Coding style](docs/agents/style.md)
2627
- [Testing patterns](docs/agents/testing.md)
2728
- [Git & PR workflow](docs/agents/git-workflow.md)

‎CONTEXT.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Reassure
2+
3+
Reassure is a performance testing toolkit that measures React and React Native scenarios repeatedly and compares measurement results between code states.
4+
5+
## Language
6+
7+
**Measurement Entry**:
8+
A single named performance scenario result produced by repeated runs of the same scenario.
9+
_Avoid_: Test stability, benchmark row
10+
11+
**Coefficient of Variation**:
12+
A unitless measure of relative duration variability for a **Measurement Entry**, calculated as standard deviation divided by mean.
13+
_Avoid_: Variation, stability
14+
15+
**Stability Check**:
16+
A workflow that compares two measurement files from the same code state to assess environment or machine stability.
17+
_Avoid_: Coefficient of variation, duration variability
18+
19+
**Run Stability**:
20+
A summary of measurement stability across all measurement entries in one measurement file.
21+
_Avoid_: Run variability, suite variability
22+
23+
**Worst Measurement Entry**:
24+
The measurement entry with the highest coefficient of variation in a measurement file.
25+
_Avoid_: Slowest test, biggest regression
26+
27+
## Relationships
28+
29+
- A **Measurement Entry** has one **Coefficient of Variation** for duration measurements.
30+
- A **Stability Check** compares two sets of **Measurement Entries**.
31+
- **Run Stability** is summarized by mean-duration-weighted average **Coefficient of Variation** and the worst **Measurement Entry** by **Coefficient of Variation**.
32+
- A comparison report may show **Run Stability** for the current and baseline measurement files separately.
33+
34+
## Example dialogue
35+
36+
> **Dev:** "Should we call `stdev / mean` test stability?"
37+
> **Domain expert:** "No. That value is the **Coefficient of Variation** for a **Measurement Entry**; a **Stability Check** compares same-code measurement files."
38+
> **Dev:** "How do we tell whether a Reassure tweak made a whole run less noisy?"
39+
> **Domain expert:** "Compare **Run Stability** using mean-duration-weighted average **Coefficient of Variation**, then inspect the worst entry."
40+
> **Dev:** "Does the worst entry mean the slowest test?"
41+
> **Domain expert:** "No. The **Worst Measurement Entry** is the entry with the highest **Coefficient of Variation**, regardless of duration."
42+
43+
## Flagged ambiguities
44+
45+
- "stability" was used for both same-code comparison and per-entry measurement spread — resolved: same-code comparison is a **Stability Check**, while `stdev / mean` is the **Coefficient of Variation**.
46+
- "worst" can mean slowest, noisiest, or most regressed — resolved: for **Run Stability**, worst means highest **Coefficient of Variation**.

‎docs/agents/domain.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Domain Language
2+
3+
Use the project terms consistently when changing comparison, reporting, or measurement behavior.
4+
5+
## Measurement Entry
6+
7+
A single named performance scenario result produced by repeated runs of the same scenario.
8+
9+
Use this for one row/scenario in a `.perf` file or comparison report. Avoid calling it a benchmark row or test stability.
10+
11+
## Coefficient of Variation
12+
13+
A unitless measure of relative duration spread for one measurement entry, calculated as:
14+
15+
```txt
16+
standard deviation / mean
17+
```
18+
19+
Use this term when discussing the statistic itself. In user-facing report fields, prefer `stability` when that is the established output label.
20+
21+
## Stability Check
22+
23+
A workflow that compares two measurement files from the same code state to assess environment or machine stability.
24+
25+
Do not use this term for the coefficient of variation of one entry. A stability check is a workflow, not the statistic.
26+
27+
## Run Stability
28+
29+
A summary of measurement stability across all measurement entries in one measurement file.
30+
31+
Run stability is the mean-duration-weighted average coefficient of variation across entries. It excludes warmup runs and includes removed outliers.
32+
33+
Comparison reports may show run stability for current and baseline measurement files separately. Do not calculate a run-level stability change unless the feature explicitly asks for it.
34+
35+
## Reporting Rules
36+
37+
- Per-entry stability belongs next to each entry's baseline/current duration details.
38+
- Run stability belongs at the end of reports.
39+
- JSON output should use readable stability keys such as `current`, `baseline`, and `weightedAverage`; avoid leaking implementation names like `currentCV`.
40+
- Use `stability` for the user-facing output label, and use coefficient of variation only when explaining the math.

‎packages/compare/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
},
5353
"dependencies": {
5454
"@callstack/reassure-logger": "1.5.1",
55+
"mathjs": "^15.2.0",
5556
"ts-markdown-builder": "0.6.0",
5657
"ts-regex-builder": "^1.8.2",
5758
"zod": "^4.4.3"

‎packages/compare/src/compare.ts‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import { writeToJson } from './output/json';
1414
import { writeToMarkdown } from './output/markdown';
1515
import { errors, warnings, logError, logWarning } from './utils/logs';
1616
import { parseHeader, parseMeasureEntries } from './utils/validate';
17+
import { calculateEntryStability, calculateRunStability } from './utils/stability';
1718

1819
/**
1920
* Probability threshold for considering given difference significant.
@@ -140,9 +141,19 @@ function compareResults(current: MeasureResults, baseline: MeasureResults | null
140141
if (currentEntry && baselineEntry) {
141142
compared.push(buildCompareEntry(name, currentEntry, baselineEntry));
142143
} else if (currentEntry) {
143-
added.push({ name, type: currentEntry.type, current: currentEntry });
144+
added.push({
145+
name,
146+
type: currentEntry.type,
147+
current: currentEntry,
148+
stability: calculateEntryStability(currentEntry),
149+
});
144150
} else if (baselineEntry) {
145-
removed.push({ name, type: baselineEntry.type, baseline: baselineEntry });
151+
removed.push({
152+
name,
153+
type: baselineEntry.type,
154+
baseline: baselineEntry,
155+
stability: calculateEntryStability(undefined, baselineEntry),
156+
});
146157
}
147158
});
148159

@@ -163,6 +174,9 @@ function compareResults(current: MeasureResults, baseline: MeasureResults | null
163174
added.sort((a, b) => a.name.localeCompare(b.name));
164175
removed.sort((a, b) => a.name.localeCompare(b.name));
165176

177+
const currentStability = calculateRunStability(current);
178+
const baselineStability = baseline ? calculateRunStability(baseline) : undefined;
179+
166180
return {
167181
metadata: { current: current.metadata, baseline: baseline?.metadata },
168182
errors,
@@ -173,6 +187,10 @@ function compareResults(current: MeasureResults, baseline: MeasureResults | null
173187
renderIssues,
174188
added,
175189
removed,
190+
stability: {
191+
current: currentStability,
192+
baseline: baselineStability,
193+
},
176194
};
177195
}
178196

@@ -197,6 +215,7 @@ function buildCompareEntry(name: string, current: MeasureEntry, baseline: Measur
197215
type: current.type,
198216
baseline,
199217
current,
218+
stability: calculateEntryStability(current, baseline),
200219
durationDiff,
201220
relativeDurationDiff,
202221
isDurationDiffSignificant,

‎packages/compare/src/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,11 @@ export type {
77
MeasureEntry,
88
CompareResult,
99
CompareMetadata,
10+
CompareStability,
1011
CompareEntry,
1112
AddedEntry,
1213
RemovedEntry,
14+
RunStability,
15+
EntryStability,
1316
RenderIssues,
1417
} from './types';

‎packages/compare/src/output/console.ts‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
import * as logger from '@callstack/reassure-logger';
2-
import type { AddedEntry, CompareResult, CompareEntry, RemovedEntry } from '../types';
3-
import { formatCount, formatDuration, formatMetadata, formatCountChange, formatDurationChange } from '../utils/format';
2+
import type { AddedEntry, CompareResult, CompareEntry, RemovedEntry, EntryStability } from '../types';
3+
import {
4+
formatCount,
5+
formatDuration,
6+
formatMetadata,
7+
formatPercent,
8+
formatCountChange,
9+
formatDurationChange,
10+
} from '../utils/format';
411
import type { MeasureMetadata } from '../types';
512

613
export function printToConsole(data: CompareResult) {
@@ -46,19 +53,29 @@ export function printToConsole(data: CompareResult) {
4653
logger.log(' - (none)');
4754
}
4855

56+
logger.log('\n➡️ Stability');
57+
printStability(data);
58+
4959
logger.newLine();
5060
}
5161

5262
function printMetadata(name: string, metadata?: MeasureMetadata) {
5363
logger.log(` - ${name}: ${formatMetadata(metadata)}`);
5464
}
5565

66+
function printStability(data: CompareResult) {
67+
const current = formatPercent(data.stability.current.weightedAverage);
68+
const baseline = data.stability.baseline ? `${formatPercent(data.stability.baseline.weightedAverage)} => ` : '';
69+
70+
logger.log(` - Weighted Average: ${baseline}${current}`);
71+
}
72+
5673
function printRegularLine(entry: CompareEntry) {
5774
logger.log(
5875
` - ${entry.name} [${entry.type}]: ${formatDurationChange(entry)} | ${formatCountChange(
5976
entry.current.meanCount,
6077
entry.baseline.meanCount
61-
)}`
78+
)} | stability ${formatEntryStability(entry.stability)}`
6279
);
6380
}
6481

@@ -81,25 +98,40 @@ function printRenderIssuesLine(entry: CompareEntry | AddedEntry) {
8198
function printAddedLine(entry: AddedEntry) {
8299
const { current } = entry;
83100
logger.log(
84-
` - ${entry.name} [${entry.type}]: ${formatDuration(current.meanDuration)} | ${formatCount(current.meanCount)}`
101+
` - ${entry.name} [${entry.type}]: ${formatDuration(current.meanDuration)} | ${formatCount(
102+
current.meanCount
103+
)} | stability ${formatEntryStability(entry.stability)}`
85104
);
86105
}
87106

88107
function printRemovedLine(entry: RemovedEntry) {
89108
const { baseline } = entry;
90109
logger.log(
91-
` - ${entry.name} [${entry.type}]: ${formatDuration(baseline.meanDuration)} | ${formatCount(baseline.meanCount)}`
110+
` - ${entry.name} [${entry.type}]: ${formatDuration(baseline.meanDuration)} | ${formatCount(
111+
baseline.meanCount
112+
)} | stability ${formatEntryStability(entry.stability)}`
92113
);
93114
}
94115

95-
export function formatInitialUpdates(count: number) {
116+
function formatEntryStability(stability: EntryStability) {
117+
if (stability.baseline != null && stability.current != null) {
118+
return `${formatPercent(stability.baseline)} → ${formatPercent(stability.current)}`;
119+
}
120+
121+
if (stability.current != null) return formatPercent(stability.current);
122+
if (stability.baseline != null) return formatPercent(stability.baseline);
123+
124+
return '?';
125+
}
126+
127+
function formatInitialUpdates(count: number) {
96128
if (count === 0) return '-';
97129
if (count === 1) return '1 initial update 🔴';
98130

99131
return `${count} initial updates 🔴`;
100132
}
101133

102-
export function formatRedundantUpdates(redundantUpdates: number[]) {
134+
function formatRedundantUpdates(redundantUpdates: number[]) {
103135
if (redundantUpdates.length === 0) return '-';
104136
if (redundantUpdates.length === 1) return `1 redundant update (${redundantUpdates.join(', ')}) 🔴`;
105137

‎packages/compare/src/output/markdown.ts‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -98,9 +98,28 @@ function buildMarkdown(data: CompareResult) {
9898
buildDetailsTable(data.removed),
9999
];
100100

101+
doc = [
102+
...doc, //
103+
md.heading('Stability', { level: 3 }),
104+
buildStabilitySection(data),
105+
];
106+
101107
return md.joinBlocks(doc);
102108
}
103109

110+
function buildStabilitySection(data: CompareResult) {
111+
return md.table(
112+
['Name', 'Baseline', 'Current'],
113+
[
114+
[
115+
'Weighted Average',
116+
data.stability.baseline ? formatPercent(data.stability.baseline.weightedAverage) : '-',
117+
formatPercent(data.stability.current.weightedAverage),
118+
],
119+
]
120+
);
121+
}
122+
104123
function buildSummaryTable(entries: Array<CompareEntry | AddedEntry | RemovedEntry>, options?: { open?: boolean }) {
105124
if (!entries.length) return md.italic('There are no entries');
106125

@@ -146,8 +165,8 @@ function formatEntryCount(entry: CompareEntry | AddedEntry | RemovedEntry) {
146165

147166
function buildDurationDetailsEntry(entry: CompareEntry | AddedEntry | RemovedEntry) {
148167
return md.joinBlocks([
149-
entry.baseline != null ? buildDurationDetails('Baseline', entry.baseline) : '',
150-
entry.current != null ? buildDurationDetails('Current', entry.current) : '',
168+
entry.baseline != null ? buildDurationDetails('Baseline', entry.baseline, entry.stability.baseline) : '',
169+
entry.current != null ? buildDurationDetails('Current', entry.current, entry.stability.current) : '',
151170
]);
152171
}
153172

@@ -158,7 +177,7 @@ function buildCountDetailsEntry(entry: CompareEntry | AddedEntry | RemovedEntry)
158177
]);
159178
}
160179

161-
function buildDurationDetails(title: string, entry: MeasureEntry) {
180+
function buildDurationDetails(title: string, entry: MeasureEntry, stability?: number) {
162181
const relativeStdev = entry.stdevDuration / entry.meanDuration;
163182

164183
return joinLines([
@@ -168,6 +187,7 @@ function buildDurationDetails(title: string, entry: MeasureEntry) {
168187
entry.durations ? `Runs: ${formatRunDurations(entry.durations)}` : '',
169188
entry.warmupDurations ? `Warmup runs: ${formatRunDurations(entry.warmupDurations)}` : '',
170189
entry.outlierDurations ? `Removed outliers: ${formatRunDurations(entry.outlierDurations)}` : '',
190+
stability != null ? `Stability: ${formatPercent(stability)}` : '',
171191
]);
172192
}
173193

0 commit comments

Comments
 (0)