Skip to content

Commit 107b5a3

Browse files
nicklafleurnicklafleur
authored
feat: Mutation caching and transitive dependency tracking (#509)
* feat: per-function hashing for incremental cache invalidation When a source file changes, only re-test mutants in functions whose AST hash changed; preserve prior results for unchanged functions in the same file. - compute_function_hashes / _compute_mutated_function_hashes in file_mutation.py: class-qualified mangled keys (x_foo / xǁClassǁmethod) -> 12-char sha256 of the function AST. Methods and nested-class methods are indexed under the same key the merge looks up, closing the latent silent-preservation bug for changed methods. - mutate_file_contents returns a 3-tuple (code, names, hashes). - SourceFileMutationData gains hash_by_function_name, persisted in .meta with a pop-with-default so old files still load. - create_mutants_for_file: mtime short-circuit now preserves all prior results instead of resetting them; on a real change, load-and-merge compares new hashes against old, resets only changed/unhashed mutants, and preserves the rest. - Tests: update all mutate_file_contents unpack sites; add tests for hash stability, body-change detection, comment-insensitivity, method key inclusion, two-function preserve/reset integration, and the method regression guard. * feat: cross-call dependency tracking for incremental stats invalidation Records caller->callee edges at stats collection time so stale outgoing call edges can be cleared when a callee's code changes. - state.py: MutmutState singleton holding old_function_hashes, current_function_hashes, and function_dependencies (callee → callers). - core.py: MutmutCallStack ContextVar propagates caller context through call chains. - trampoline.py stats branch: resolves caller via MutmutCallStack, passes it to record_trampoline_hit, sets updated context for inner calls, respects MUTMUT_DEPENDENCY_DEPTH env ceiling. - record_trampoline_hit gains caller param; upstream's source-path- resolving max_stack_depth walk preserved verbatim; dependency edge written only when track_dependencies=True. - FileMutationResult gains changed_functions/current_hashes (deferred from commit 1); create_mutants accumulates current_hashes into state().current_function_hashes across worker results. - create_mutants_for_file builds module-qualified current_hashes and changed_functions for return to parent. - load_stats/save_stats persist function_hashes and function_dependencies alongside existing test associations (backwards-compatible pop-with- default on load). - _cleanup_stale_stats: removes test associations and dependency edges for modules absent from current_function_hashes. - _invalidate_stale_dependency_edges: clears changed functions from all caller sets so stale outgoing edges are rebuilt on next stats run. - collect_or_load_stats: on incremental load, runs cleanup always and invalidation when track_dependencies; persists the result. - Config gains track_dependencies (default True) and dependency_tracking_depth (default None); run_stats_collection sets MUTMUT_DEPENDENCY_DEPTH from config. - Tests: record_trampoline_hit with/without track_dependencies, _cleanup_stale_stats removes unknown modules, _invalidate_stale_ dependency_edges clears changed callers and no-ops on first run, config defaults asserted. * e2e: add benchmark project with 1k mutants - Add e2e_projects/benchmark_1k/ with ~1000 mutants for testing - Includes modules: numbers, strings, booleans, operators, comparisons, arguments, returns, complex (recursion, higher-order functions) - Configurable delays via BENCHMARK_IMPORT_DELAY, BENCHMARK_CONFTEST_DELAY, BENCHMARK_TEST_DELAY environment variables to simulate the performance under variable test and startup runtimes. * feat: invalidate cache on config and dependency changes Cached verdicts were only invalidated when a function body changed, so changes to config or dependency files silently produced stale results. - Config.config_fingerprint() hashes result-affecting config, grouped so we reset only what each change can affect: - timeout change -> reset only timeout verdicts - type_check_command change -> reset mutants whose type-check status flips (symmetric difference of old exit-37 and newly-caught) - pytest_add_cli_args / test-selection change -> reset all results and force full stats recollection - set-affecting config (source_paths, only_mutate, ...) is ignored: new mutants are uncached and dropped ones stop being walked - compute_watched_file_hashes() hashes dependency/build files (pyproject.toml, setup.cfg/py, requirements*.txt, lockfiles) plus user globs from the new cache_invalidation_files config. The on_dependency_change config ("warn" | "rerun" | "ignore", default "warn") controls whether a change warns or resets all results. - Fingerprints persist in mutmut-stats.json with pop-with-default, so old caches load and a missing fingerprint triggers no invalidation. * feat: use git to detect non-Python dependency file changes Replace the fixed watched-file list with git-based change detection. mutmut now uses `git diff`/`git ls-files` to find every non-.py file changed since the last full run, falling back to the curated list when git is unavailable. A default exclude set (*.md, *.rst, docs/, LICENSE, etc.) drops files that never affect tests; users can extend it with `cache_invalidation_exclude`. The git commit and file hashes are persisted together as a baseline so a later git-less environment (e.g. a separate CI stage) can still detect changes to previously-tracked files by re-hashing them. New options: `use_git_change_detection` (default true) and `cache_invalidation_exclude`. * HISTORY * lock * fix: address three cache-correctness bugs from Copilot review - Return cached function hashes for mtime-skipped files so _cleanup_stale_stats and _invalidate_stale_dependency_edges don't treat unchanged files as deleted; use get_mutant_name at both call sites instead of inlining the path→module conversion - Change dependency_tracking_depth default from None to -1 so setup.cfg values are correctly coerced to int by the config loader; narrow type from int | None to int and drop the conditional in run_stats_collection - Fix benchmark fixture to yield unconditionally so it doesn't fail when BENCHMARK_TEST_DELAY=0
1 parent bd6057f commit 107b5a3

41 files changed

Lines changed: 4598 additions & 41 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎HISTORY.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,22 @@
11
Changelog
22
---------
33

4+
Unreleased
5+
~~~~~~~~~~
6+
7+
* Per-function source hashing for incremental cache invalidation — only re-test mutants in functions that changed
8+
9+
* Cross-call dependency tracking — invalidate mutants in callers when a called function changes
10+
11+
* Use git to detect non-Python dependency file changes; falls back to a curated file list when git is unavailable
12+
13+
* Add `cache_invalidation_exclude` config to suppress noisy files from change detection
14+
15+
* Add `use_git_change_detection` config (default true) to opt out of git-based detection
16+
17+
* Invalidate cached results automatically when result-affecting config fields change
18+
19+
420
3.6.0
521
~~~~~
622

‎README.rst‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -401,6 +401,77 @@ You can add and override pytest arguments:
401401
also_copy = ["mutmut_pytest.ini"]
402402
403403
404+
Detecting dependency and config changes
405+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
406+
407+
Between runs, mutmut only re-tests mutants in functions whose source changed.
408+
Changes outside your Python source — a dependency upgrade, a data file, a
409+
config file — cannot be tied to a function, so they would otherwise be missed
410+
and you would get cached results that no longer reflect reality.
411+
412+
To catch this, mutmut detects non-Python files that changed since the last full
413+
run and warns you about them. If your project is a git repository and git is
414+
installed, mutmut uses git (a soft dependency no extra package is required) to
415+
find every changed non-Python file, respecting your `.gitignore`. Python files
416+
are excluded because their changes are already tracked per function.
417+
418+
On a full run with git available, mutmut also records the content hashes of the
419+
tracked non-Python files. This means a later run in an environment without git
420+
(for example a different CI stage) can still detect changes to that known set of
421+
files, even though it cannot discover brand-new ones.
422+
423+
When git is unavailable, mutmut falls back to hashing a curated set of build and
424+
dependency files:
425+
426+
- `pyproject.toml`
427+
- `setup.cfg`
428+
- `setup.py`
429+
- `requirements*.txt`
430+
- `poetry.lock`
431+
- `uv.lock`
432+
- `Pipfile`
433+
- `Pipfile.lock`
434+
435+
You can watch additional files (for example data files your tests depend on)
436+
with the `cache_invalidation_files` config, which accepts glob patterns
437+
resolved against the project root. These are checked even when git ignores them,
438+
and are never dropped by the exclusions below:
439+
440+
.. code-block:: toml
441+
442+
cache_invalidation_files = [ "queries/*.sql", "config/*.yaml" ]
443+
444+
Git detection reports every changed non-Python file, so mutmut drops files that
445+
practically never affect tests (markdown, `LICENSE`, `CHANGELOG`, `docs/`, git
446+
and editor metadata, ...). Exclude additional noisy files with
447+
`cache_invalidation_exclude` (glob patterns, `*` spans directories):
448+
449+
.. code-block:: toml
450+
451+
cache_invalidation_exclude = [ "*.json", "fixtures/snapshots/*" ]
452+
453+
When a watched file changes, `on_dependency_change` controls what happens:
454+
455+
- `warn` (default): list the changed files and keep the cache.
456+
- `rerun`: re-test all mutants.
457+
- `ignore`: do nothing.
458+
459+
.. code-block:: toml
460+
461+
on_dependency_change = "warn"
462+
463+
Git detection is on by default; disable it (forcing the curated-list fallback)
464+
with:
465+
466+
.. code-block:: toml
467+
468+
use_git_change_detection = false
469+
470+
Changes to mutmut's own result-affecting config (such as `pytest_add_cli_args`,
471+
`type_check_command`, or the timeout settings) are always detected and
472+
invalidate the affected cached results automatically.
473+
474+
404475
Unstable configs
405476
~~~~~~~~~~~~~~~~
406477

‎e2e_projects/benchmark_1k/README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Benchmark 1K
2+
3+
A synthetic benchmark project with 1000 mutants for validating mutmut's process isolation and hot-fork warmup strategy performance.
4+
5+
**TL;DR:**
6+
- `fork` is fastest and nearly immune to import delays (requires fork-safe libraries)
7+
- `collect` (hot-fork default) is 2-9x faster than `import`/`none` depending on import cost
8+
- Higher import delays dramatically penalize `import` and `none` strategies
9+
10+
11+
## Mutant Distribution
12+
13+
| Type | Total | Killed | Survived | Kill Rate |
14+
|------------|-------|--------|----------|-----------|
15+
| return | 221 | 161 | 60 | 73% |
16+
| number | 159 | 99 | 60 | 62% |
17+
| argument | 141 | 132 | 9 | 94% |
18+
| string | 125 | 78 | 47 | 62% |
19+
| boolean | 120 | 47 | 73 | 39% |
20+
| comparison | 119 | 19 | 100 | 16% |
21+
| operator | 115 | 90 | 25 | 78% |
22+
| **Total** | **1000** | **626** | **374** | **63%** |
23+
24+
## Usage
25+
26+
### Run mutation testing
27+
28+
```bash
29+
cd e2e_projects/benchmark_1k
30+
mutmut run
31+
```
32+
33+
### Run benchmark comparison
34+
35+
```bash
36+
python run_benchmark.py
37+
```
38+
39+
This runs `mutmut run` under each strategy (`fork`, `collect`, `import`, `none`) and outputs:
40+
- Throughput (mutations/second) for each strategy
41+
- Results saved to `benchmark_results.json`
42+
43+
### View results
44+
45+
```bash
46+
cat mutants/summary.json | python -m json.tool
47+
```
48+
49+
## Test Design
50+
51+
Tests are fast unit tests with instant assertions. Configurable delays simulate real-world costs:
52+
53+
- **Import delay**: Simulates library import time (Flask, SQLAlchemy, etc.)
54+
- **Conftest delay**: Simulates fixture/plugin setup time
55+
- **Test delay**: Per-test runtime with +/-10% gaussian jitter for realistic variance
56+
57+
Usage:
58+
```bash
59+
python run_benchmark.py --test-delay 0.01 # Add 10ms per-test with jitter
60+
```

0 commit comments

Comments
 (0)