-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add quality modes and management docs #8732
Conversation
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe pull request includes several modifications across three documents related to quality feedback and control in the CVAT system. The "Immediate job feedback" document has been refined for clarity and consistency, while the "Automated QA, Review & Honeypots" document has been significantly restructured to enhance understanding of annotation quality evaluation. Additionally, a new document titled "quality-control.md" has been introduced, providing a comprehensive overview of quality control features, including validation sets and job validation processes. Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (8)
site/content/en/docs/manual/basics/quality-control.md (5)
14-15
: Add a comma for better readability.-In this section we only highlight the key steps in quality estimation. +In this section, we only highlight the key steps in quality estimation.🧰 Tools
🪛 LanguageTool
[typographical] ~14-~14: It appears that a comma is missing.
Context: ...to-check-task-quality-metrics) In this section we only highlight the key steps in qual...(DURING_THAT_TIME_COMMA)
34-35
: Enhance the formatting of the limitation notice.The limitation about Ground Truth mode should be more prominent. Consider using a warning or note admonition.
-> For already existing tasks only the Ground Truth validation mode is available. If you want -> to use Honeypots for your task, you will need to recreate the task. +> **Warning** +> For already existing tasks, only the Ground Truth validation mode is available. If you want +> to use Honeypots for your task, you will need to recreate the task.🧰 Tools
🪛 LanguageTool
[uncategorized] ~34-~34: A comma might be missing here.
Context: ...y existing task > For already existing tasks only the Ground Truth validation mode i...(AI_EN_LECTOR_MISSING_PUNCTUATION_COMMA)
54-54
: Fix typo in configuration guidance.-1. Set **Max validations per job** to above zero. 3 is a good staring number. +1. Set **Max validations per job** to above zero. 3 is a good starting number.
70-72
: Consider adding more details about quality metrics.The instructions would be more helpful with examples of:
- What metrics are available in the summaries
- What to look for in detailed reports
- Typical completion time for metrics computation
Would you like me to help draft additional content for this section?
84-84
: Consider adding a troubleshooting section.The documentation would benefit from a section addressing common issues and their solutions, such as:
- What to do if validation fails
- How to handle conflicts in annotations
- Steps to take when quality metrics are below expectations
Would you like me to help draft a troubleshooting section?
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md (3)
14-14
: Fix grammar and style issues for better readability.Several grammar and style issues need to be addressed:
- Replace "2d" with "2D" for proper capitalization
- Fix "this methods" to "this method"
- Add missing commas after introductory phrases
-that quality estimation only supports 2d tasks +that quality estimation only supports 2D tasks -Note that using this methods can +Note that using this method can -In this mode some of the task frames +In this mode, some of the task frames -For each frame you can +For each frame, you canAlso applies to: 217-217, 190-190, 226-226, 287-287, 292-292
🧰 Tools
🪛 LanguageTool
[grammar] ~14-~14: Did you mean “2D”(= two-dimensional) or “2nd” (= second)?
Context: ...that quality estimation only supports > 2d tasks. It supports all the annotation t...(THREE_D)
[grammar] ~14-~14: Did you mean “2D”(= two-dimensional) or “2nd” (= second)?
Context: ...upports all the annotation types except 2d cuboids. > Note that tracks are co...(THREE_D)
183-186
: Enhance the explanation of validation modes.The distinction between Ground Truth and Honeypots modes could be clearer. Consider adding a brief comparison table or diagram at the beginning of this section to help users quickly understand the key differences before diving into the details.
383-385
: Consider moving premium feature notice to a more prominent location.The note about quality analytics being a premium feature is buried in the middle of the document. Consider moving this important information to the beginning of the document or adding a "Prerequisites" section that clearly outlines what features require premium access.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (5)
site/content/en/images/honeypot09.jpg
is excluded by!**/*.jpg
site/content/en/images/honeypot10.jpg
is excluded by!**/*.jpg
site/content/en/images/honeypot11.jpg
is excluded by!**/*.jpg
site/content/en/images/honeypot12.jpg
is excluded by!**/*.jpg
site/content/en/images/immediate-feedback-settings-validations.jpg
is excluded by!**/*.jpg
📒 Files selected for processing (3)
site/content/en/docs/enterprise/immediate-feedback.md
(2 hunks)site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
(4 hunks)site/content/en/docs/manual/basics/quality-control.md
(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- site/content/en/docs/enterprise/immediate-feedback.md
🧰 Additional context used
🪛 LanguageTool
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
[grammar] ~14-~14: Did you mean “2D”(= two-dimensional) or “2nd” (= second)?
Context: ...that quality estimation only supports > 2d tasks. It supports all the annotation t...
(THREE_D)
[grammar] ~14-~14: Did you mean “2D”(= two-dimensional) or “2nd” (= second)?
Context: ...upports all the annotation types except 2d cuboids. > Note that tracks are co...
(THREE_D)
[grammar] ~64-~64: The usual collocation for “independently” is “of”, not “from”. Did you mean “independently of”?
Context: ...ion set images are sampled randomly and independently from the full dataset. That is, for the qual...
(INDEPENDENTLY_FROM_OF)
[typographical] ~190-~190: It appears that a comma is missing.
Context: ... be managed. ### Ground Truth In this mode some of the task frames are selected in...
(DURING_THAT_TIME_COMMA)
[grammar] ~217-~217: The singular determiner ‘this’ may not agree with the plural noun ‘methods’. Did you mean “these”?
Context: ...e quality in such jobs. Note that using this methods can result in increased total s...
(THIS_NNS)
[grammar] ~217-~217: Use the singular noun “method” with the word ‘this’.
Context: ...lity in such jobs. Note that using this methods can result in increased total size of t...
(DT_JJ_NNS_MD)
[typographical] ~226-~226: It appears that a comma is missing.
Context: ...b, if possible. ### Honeypots In this mode some random frames of the task are sele...
(DURING_THAT_TIME_COMMA)
[style] ~229-~229: Consider replacing ‘only’ with a different word to let your writing stand out.
Context: ...lidation frames are used. This mode can only be used at task creation and cannot be ...
(ONLY_EXCLUSIVELY_STYLE)
[formatting] ~267-~267: Consider inserting a comma after an introductory phrase for better readability.
Context: ...xample: a video annotation with tracks. In this case there is only 1 option - the Ground Tru...
(IN_THAT_CASE_COMMA)
[style] ~270-~270: Consider an alternative for the often overused word ‘important’.
Context: ... dataset annotation, image order is not important. Here you can use both options. You can...
(NOT_IMPORTANT)
[style] ~274-~274: Specify a number, remove phrase, use “a few”, or use “some”
Context: ...validation (for instance, you have only a small number of validation frames) or you want to allow...
(SMALL_NUMBER_OF)
[typographical] ~287-~287: It appears that a comma is missing.
Context: ... and review the frame details. For each frame you can see the number of uses in the t...
(DURING_THAT_TIME_COMMA)
[typographical] ~292-~292: It appears that a comma is missing.
Context: ... #### Frame changes In both validation modes it's possible to exclude some of the va...
(DURING_THAT_TIME_COMMA)
[grammar] ~294-~294: Did you mean “if” or “is” (with a reflexive pronoun)?
Context: ... validation frames are "bad", extra, or it they have incorrect annotations, which ...
(CC_IT_PRP)
[grammar] ~331-~331: The word “backup” is a noun. The verb is spelled with a space.
Context: ... on the honeypot frames. If you want to backup annotations, > use a task backup or exp...
(NOUN_VERB_CONFUSION)
[uncategorized] ~396-~396: Did you mean “if”?
Context: ...ated immediately** after task updates. It you want request quality metrics update...
(IT_IF_PREMIUM)
site/content/en/docs/manual/basics/quality-control.md
[typographical] ~14-~14: It appears that a comma is missing.
Context: ...to-check-task-quality-metrics) In this section we only highlight the key steps in qual...
(DURING_THAT_TIME_COMMA)
[uncategorized] ~34-~34: A comma might be missing here.
Context: ...y existing task > For already existing tasks only the Ground Truth validation mode i...
(AI_EN_LECTOR_MISSING_PUNCTUATION_COMMA)
🔇 Additional comments (2)
site/content/en/docs/manual/basics/quality-control.md (1)
1-6
: LGTM! Front matter is well-structured.
The metadata is properly formatted and includes all necessary fields for documentation organization.
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md (1)
167-167
: Verify API references and update instructions.
A few items need verification:
- The link to Job API create() should be verified to ensure it's current
- The refresh process for quality metrics could benefit from more detailed steps or screenshots
Also applies to: 396-398
✅ Verification successful
Let me gather more information about the API documentation to verify the links and content.
Documentation appears complete with proper references and visual aids
The verification shows that:
- The API reference link is consistently used in the documentation
- The quality metrics refresh process is well documented with:
- Clear instructions for manual refresh
- A screenshot showing the refresh button location
- Visual guides for reviewing the metrics
- Explanation of computation time and requirements
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Verify API documentation links
rg -l "Job API create\(\)"
Length of output: 101
Script:
#!/bin/bash
# Check for API documentation files and references
fd -e md -e rst -e yaml api_sdk
rg -l "api_sdk/sdk/reference/apis/jobs-api"
Length of output: 150
Script:
#!/bin/bash
# Look for quality metrics documentation
rg -A 5 "quality metrics" "site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md"
Length of output: 941
Script:
#!/bin/bash
# Check if there are any screenshots or image references in the auto-qa documentation
rg -i "!\[.*\]" "site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md"
Length of output: 707
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
|
||
Import, export, and delete options are available from the | ||
job's menu. | ||
## Validation modes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
site/content/en/images/immediate-feedback-settings-validations.jpg
Outdated
Show resolved
Hide resolved
Co-authored-by: Maria Khrustaleva <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM 👍
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Outdated
Show resolved
Hide resolved
site/content/en/images/immediate-feedback-settings-validations.jpg
Outdated
Show resolved
Hide resolved
site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
Show resolved
Hide resolved
Co-authored-by: Maria Khrustaleva <[email protected]>
|
<!-- Raise an issue to propose your change (https://github.com/cvat-ai/cvat/issues). It helps to avoid duplication of efforts from multiple independent contributors. Discuss your ideas with maintainers to be sure that changes will be approved and merged. Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/). --> <!-- Provide a general summary of your changes in the Title above --> ### Motivation and context <!-- Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. Describe your changes in detail, add screenshots. --> Fixes invalid extension of an image changed in #8732 ### How has this been tested? <!-- Please describe in detail how you tested your changes. Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ### Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole line. If you don't do that, GitHub will show incorrect progress for the pull request. If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [ ] I submit my changes into the `develop` branch - [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md --> - [ ] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] I have linked related issues (see [GitHub docs]( https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)) - [ ] I have increased versions of npm packages if it is necessary ([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning), [cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning), [cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning) and [cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning)) ### License - [ ] I submit _my code changes_ under the same [MIT License]( https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
Motivation and context
How has this been tested?
Checklist
develop
branch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit