[GSoC'24] added documentation for consensus feature #8401
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Viditagarwal7479
changed the title
| The basic use case is annotating the entire dataset with multiple annotators to obtain high-quality annotations in the dataset. In some cases it may work, but typically, however, this way of annotating is prohibitively expensive, as you need to annotate everything several times. There are several ways how the situation can be improved. | ||
|
|
||
| 1. You can use several highly skilled annotators and rely on their consensus to create high-quality Ground Truth annotations. Typically, the Ground Truth subset contains a small portion of the whole dataset (e.g. 2-5%), so it's relatively easy to annotate it several times. These annotations are used then to validate work of different annotators randomly, so it's very desirable to avoid single expert bias in Ground Truth, and this is one of the key benefits of the consensus annotation. This method requires extra work, but it's much lower than annotating all the dataset several times. For instance, annotating a 3% of images 5 times will result in extra 3% * (5 - 1) = 12% overhead, comparing to single GT annotations. | ||
| 2. In cases of points and poly-line / poly-shape annotations, annotators can add more annotations without a fixed structure, which can introduce significant variation. Similarly, for skeleton annotations, while there is a structure, annotators can move the points around. Due to the presence of numerous points, this can lead to significant variations in annotations for the same data. Using a consensus-based approach for such data types can help reduce these variations, which may arise from minor errors in placing the points. |
There was a problem hiding this comment.
Actually, this use case is not covered completely yet.
There was a problem hiding this comment.
as while merging we are using their bbox representation, instead of updating the algorithm?
There was a problem hiding this comment.
As we're only using one of the input shapes as a whole and have not implemented per annotation element merging.
There was a problem hiding this comment.
Nice, makes sense
There was a problem hiding this comment.
So, would it better to remove this use case, or let it be here as aparaently that's a use case for this feature. Though in a toned down sense?
Viditagarwal7479
requested review from
azhavoro,
bsekachev,
SpecLad,
Marishka17 and
nmanovic
as code owners
|
#7973
Adding documentation for consensus features
Checklist
- [ ] I submit my changes into thedevelopbranch- [ ] I have created a changelog fragment- [ ] I have added tests to cover my changes- [ ] I have increased versions of npm packages if it is necessary(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.