Improve error messages for ast-grep test failures based on failure type #2174

Copilot · 2025-08-11T22:06:25Z

This PR improves the error messages shown when ast-grep test fails by providing context-aware help based on the type of failures encountered.

Problem

Previously, when ast-grep test failed, users would always see a generic message directing them to the playground:

Error: test failed. 0 passed; 1 failed;
Help: You can use ast-grep playground to debug your rules and test cases.
See also: https://ast-grep.github.io/playground.html

However, this wasn't helpful when the rule was actually working correctly but the snapshot tests just needed to be updated. Users would waste time debugging rules that didn't need fixing.

Solution

The implementation now analyzes the types of test failures and provides appropriate guidance:

When all failures are snapshot mismatches:

Error: test failed. 0 passed; 1 failed;
Help: Test failed due to mismatching snapshots. Run with `--update-all` to update all snapshots, or `--interactive` to update selectively.
See also: https://ast-grep.github.io/guide/test-rule.html

When there are mixed failures or non-snapshot failures:

Error: test failed. 0 passed; 1 failed;
Help: You can use ast-grep playground to debug your rules and test cases.
See also: https://ast-grep.github.io/playground.html

Implementation Details

Clean enum design: Redesigned TestResult enum with Success, RuleFail, and MismatchSnapshotOnly variants with proper separation of concerns
High-performance failure analysis: Optimized CaseResult::is_snapshot_mismatch_only_failure() to use a single loop instead of multiple iterator chains with allocations
Contextual error messages: Enhanced messages to explain the reason for failure ("Test failed due to mismatching snapshots") while providing actionable next steps
Clean architecture: Moved ErrorContext creation to verify.rs where it's consumed, improving code organization

Testing

The change includes comprehensive unit tests covering snapshot-only failures, mixed failures, and successful test scenarios. All existing tests continue to pass, ensuring backward compatibility.

Fixes #2173.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

codecov · 2025-08-11T22:19:47Z

Codecov Report

❌ Patch coverage is 96.15385% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.19%. Comparing base (4e28f4d) to head (cfc3ff9).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/cli/src/utils/error_context.rs	0.00%	3 Missing ⚠️
crates/cli/src/verify.rs	72.72%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2174      +/-   ##
==========================================
+ Coverage   86.10%   86.19%   +0.09%     
==========================================
  Files         107      107              
  Lines       16865    17012     +147     
==========================================
+ Hits        14522    14664     +142     
- Misses       2343     2348       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: HerringtonDarkholme <[email protected]>

HerringtonDarkholme

@copilot please address the code feedback

HerringtonDarkholme · 2025-08-11T22:34:03Z

crates/cli/src/verify/reporter.rs

  }
  /// A hook function runs after tests completed.
-  fn after_report(&mut self, results: &[CaseResult]) -> Result<(bool, String)> {
+  fn after_report(&mut self, results: &[CaseResult]) -> Result<(bool, ErrorContext)> {


This should not return (bool, ErrorContext). If the test succeed there is no error.
Please design a proper data type for test result

Fixed in commit 988729f. Replaced the (bool, ErrorContext) return type with a proper TestResult enum that has Success and Failure variants. This eliminates the confusion between success and error states.

HerringtonDarkholme · 2025-08-11T22:42:57Z

crates/cli/src/verify/reporter.rs

        passed += 1;
      } else {
        failed += 1;
+        for status in &result.cases {


Can you move this to result struct? Give it a name like is_snapshot_mismatch_only_failure also add comment why this method exists

Fixed in commit 988729f. Moved the logic to CaseResult::is_snapshot_mismatch_only_failure() method with documentation explaining its purpose for determining appropriate error messages when test failures occur.

HerringtonDarkholme · 2025-08-11T22:44:03Z

crates/cli/src/utils/error_context.rs

  WriteFile(PathBuf),
  // Test
  TestFail(String),
+  TestFailSnapshotMismatch(String),


Pleae change it to TestSnapshotMismatch for a shorter name.

Fixed in commit 988729f. Renamed TestFailSnapshotMismatch to TestSnapshotMismatch for a more concise name.

HerringtonDarkholme · 2025-08-11T22:44:40Z

crates/cli/src/utils/error_context.rs

      DiagnosticError(_) => 1,
      ProjectNotExist | LanguageNotSpecified | RuleNotSpecified | RuleNotFound(_) => 2,
      TestFail(_) => 3,
+      TestFailSnapshotMismatch(_) => 3,


Please merge this test to the line above using |

Fixed in commit 988729f. Merged the test cases using TestFail(_) | TestSnapshotMismatch(_) => 3 for cleaner code.

HerringtonDarkholme · 2025-08-11T22:46:05Z

crates/cli/src/utils/error_context.rs

      ),
+      TestFailSnapshotMismatch(message) => Self::new(
+        message,
+        "It looks like your rule is working correctly but the snapshot test needs to be updated. You can run the test with `--update-all` to update all snapshots, or use `--interactive` to selectively update snapshots.",


The message is too verbose. Please change it in the same style as other error message. Stick to the philosophy mentioned in the comment.

// * Use consistent terminology // * Be clear and concise // * Provide context // * Suggest a fix

Fixed in commit 988729f. Simplified the error message to "Run with --update-all to update all snapshots, or --interactive to update selectively." This follows the existing style guidelines: clear, concise, provides context, and suggests a fix.

coderabbitai · 2025-08-11T22:46:54Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Join our Discord community for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

…ssages - Replace (bool, ErrorContext) return type with proper TestResult enum - Move snapshot mismatch detection logic to CaseResult::is_snapshot_mismatch_only_failure method with documentation - Rename TestFailSnapshotMismatch to TestSnapshotMismatch for brevity - Merge test cases using | operator where appropriate - Simplify error message to be more concise and consistent with existing style - Add comprehensive unit tests for new functionality Co-authored-by: HerringtonDarkholme <[email protected]>

HerringtonDarkholme

@copilot please address comment

crates/cli/src/verify/case_result.rs

HerringtonDarkholme · 2025-08-12T12:38:10Z

crates/cli/src/utils/error_context.rs

      ),
+      TestSnapshotMismatch(message) => Self::new(
+        message,
+        "Run with `--update-all` to update all snapshots, or `--interactive` to update selectively.",


Please add context like "test failed due to mismatching snapshots"

Fixed in commit a009b97. Added contextual information "Test failed due to mismatching snapshots" to provide clear reason for the failure.

crates/cli/src/verify/reporter.rs

…messages Co-authored-by: HerringtonDarkholme <[email protected]>

HerringtonDarkholme

@copilot please address comment

crates/cli/src/verify.rs

crates/cli/src/verify/reporter.rs

Co-authored-by: HerringtonDarkholme <[email protected]>

Copilot · 2025-08-12T13:57:57Z

@copilot please address comment

Addressed all feedback in commit cfc3ff9. The TestResult enum no longer contains ErrorContext - it's now built in verify.rs where it's used, providing cleaner separation of concerns.

Initial plan

9378bde

Copilot AI assigned Copilot and HerringtonDarkholme Aug 11, 2025

Copilot started work on behalf of HerringtonDarkholme August 11, 2025 22:06 View session

Implement improved error messages for test failures

763d841

Co-authored-by: HerringtonDarkholme <[email protected]>

Copilot AI changed the title ~~[WIP] [feature] improve error message for ast-grep test failure~~ Improve error messages for ast-grep test failures based on failure type Aug 11, 2025

Copilot AI requested a review from HerringtonDarkholme August 11, 2025 22:23

Copilot finished work on behalf of HerringtonDarkholme August 11, 2025 22:23

HerringtonDarkholme requested changes Aug 11, 2025

View reviewed changes

HerringtonDarkholme marked this pull request as ready for review August 11, 2025 22:46

Copilot started work on behalf of HerringtonDarkholme August 11, 2025 22:46 View session

Copilot AI requested a review from HerringtonDarkholme August 11, 2025 23:02

Copilot finished work on behalf of HerringtonDarkholme August 11, 2025 23:02

HerringtonDarkholme requested changes Aug 12, 2025

View reviewed changes

Copilot started work on behalf of HerringtonDarkholme August 12, 2025 12:41 View session

Address code review feedback: optimize performance and improve error …

a009b97

…messages Co-authored-by: HerringtonDarkholme <[email protected]>

Copilot AI requested a review from HerringtonDarkholme August 12, 2025 12:52

Copilot finished work on behalf of HerringtonDarkholme August 12, 2025 12:52

HerringtonDarkholme requested changes Aug 12, 2025

View reviewed changes

crates/cli/src/verify.rs Outdated Show resolved Hide resolved

crates/cli/src/verify/reporter.rs Outdated Show resolved Hide resolved

Copilot started work on behalf of HerringtonDarkholme August 12, 2025 13:46 View session

Remove ErrorContext from TestResult enum and build it in verify.rs

cfc3ff9

Co-authored-by: HerringtonDarkholme <[email protected]>

Copilot AI requested a review from HerringtonDarkholme August 12, 2025 13:59

Copilot finished work on behalf of HerringtonDarkholme August 12, 2025 13:59

HerringtonDarkholme approved these changes Aug 15, 2025

View reviewed changes

HerringtonDarkholme merged commit 08f6fdc into main Aug 15, 2025
2 of 3 checks passed

HerringtonDarkholme deleted the copilot/fix-2173 branch August 15, 2025 04:24

Uh oh!

Improve error messages for ast-grep test failures based on failure type #2174

Improve error messages for ast-grep test failures based on failure type #2174

Uh oh!

Conversation

Copilot AI commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Implementation Details

Testing

Uh oh!

codecov bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

HerringtonDarkholme left a comment

Choose a reason for hiding this comment

Uh oh!

HerringtonDarkholme Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

HerringtonDarkholme Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

HerringtonDarkholme Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

HerringtonDarkholme Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

HerringtonDarkholme Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Aug 11, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

HerringtonDarkholme left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HerringtonDarkholme Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HerringtonDarkholme left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Aug 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Copilot AI commented Aug 11, 2025 •

edited

Loading

codecov bot commented Aug 11, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)