Rollout of new Coverage engine November 23, 2023#
GitHub only: this feature requires updated app permissions. If you haven't done so yet, please review and accept the updated Codacy app permissions on GitHub
As part of an ongoing effort to improve the speed and value of the insights provided by Codacy, we've been working on a new Coverage engine and started its deployment on November 23rd, 2023. The rollout to use the new engine across Codacy will be phased across several months and will gradually impact the coverage data you see on the Git provider, UI, and API.
Please refer to the table below for the updated status of the transition process. The table will be updated as changes are introduced.
| Feature set | Feature | Status | Notes |
|---|---|---|---|
| Git providers | Git provider status checks | Live | The new Coverage engine now sends coverage data to your Git provider. Codacy stopped sending coverage checks from the old Coverage engine on June 5th, 2024. |
| GitHub coverage summaries | Live | The new Coverage engine now sends coverage summaries to GitHub. If you haven't done so yet, please review and update your app permissions as mentioned above. | |
| Other pages and components | Planned | - |
Note
Because of this transition, both old and new data will coexist during this period, potentially leading to differences in reported metrics.
Git provider status checks from the new Coverage engine#
This section applies to the repositories for which you set Codacy to send pull request coverage status data to your Git provider (see how on GitHub, GitLab, and Bitbucket).
On June 5th 2024, Codacy stopped sending coverage checks from the old Coverage engine. As a consequence of this, you will only see the new checks from the new Coverage engine, Codacy Diff Coverage and Codacy Coverage Variation, and will no longer see the old Codacy Coverage Report check on your pull requests.
Important
GitHub only: If you are using the Codacy Coverage Report status check to block merging pull requests, please update your GitHub setup to fix any disruptions.
This is what the new checks look like on each provider:
-
On GitHub:
-
On GitLab:
-
On Bitbucket:
Deprecation and removal calendar for the old Coverage engine status checks#
| Date | Event | What to expect |
|---|---|---|
| February 14th, 2024 | Deprecation of the old Coverage status check |
From this day on, the new checks will be the official status checks for Codacy Coverage and will no longer be marked beta. The old check will be marked deprecated but will keep working. If on this date you are still using the old check to block merging pull requests on GitHub, please update your Git provider settings. |
| April 18th, 2024 | Brownout of the old Coverage status check |
Codacy will not send the old status check on this day. This will help you confirm that your setup has been updated correctly and lower the risk of disruptions when the old status check is removed. |
| June 5th, 2024 | Removal of the old Coverage status check |
Codacy will stop sending the old status check. If on this date you are still using the old check to block merging pull requests on GitHub, you will no longer be able to merge them. To fix this, follow the instructions below. |
If you are using the old Coverage status check to block merging pull requests on GitHub#
If you are using the old status check to block merging pull requests on GitHub, remove the old Codacy Coverage Report as a required check of your pull requests' target branch:
- Edit the branch protection rule of your pull requests' target branch.
- Remove the old Codacy Coverage Report under the section Require status checks to pass before merging and save your changes:
GitHub coverage summaries from the new Coverage engine#
This section applies to the repositories for which you set Codacy to post coverage summaries to your GitHub pull requests
GitHub only: this feature requires updated app permissions. If you haven't done so yet, please review and accept the updated Codacy app permissions on GitHub
On December 7th 2023, the new Coverage engine started posting coverage summaries to GitHub, replacing the old Coverage engine.
New Coverage engine side effects#
Missing data on the Coverage Pull request page#
You may notice missing coverage data on the Coverage Pull requests page for older pull requests when compared to the coverage data shown on the Quality Pull requests page. This may happen due to a number of reasons:
-
GitHub only: The new Coverage service requires updated app permissions to calculate coverage data. If you haven't done so yet, review and accept the updated Codacy app permissions on GitHub and re-analyze the pull request by pushing an empty commit.
-
The data may be missing because the latest commit to the pull request was pushed before the new Coverage engine was enabled. To resolve this, re-analyze the pull request by pushing an empty commit:
git commit --allow-empty -m "Trigger analysis" git push
Differences in coverage metrics between the old and new Coverage engines#
You may notice some differences in the coverage metrics reported by the old and new Coverage engines and may need to update your coverage gate rules accordingly. This may happen because the new Coverage engine calculates coverage metrics considering all the files included in the coverage report, while the old Coverage engine ignores some files:
- The old Coverage engine ignores any files on a coverage report that aren't present on the repository on that given commit.
- The old Coverage engine may ignore additional files since it shares ignore rules with the Codacy analysis engine.
Share your feedback 📢
Did this page help you?
Thanks for the feedback! Is there anything else you'd like to tell us about this page?
255 characters left
We're sorry to hear that. Please let us know what we can improve:
255 characters left
Alternatively, you can create a more detailed issue on our GitHub repository.
Thanks for helping improve the Codacy documentation.
Edit this page on GitHub if you notice something wrong or missing.
If you have a question or need help please contact [email protected].