Red Hen Lab Rapid Annotator summary for the final evaluation of google summer of code includes the work done through 2021 and 2022 by Mohamed Mokhtar

• Introduction
• Terminologies
• Built with
• Important links
• Issues list
• Pull requests list
• Accomplished tasks list
• Challenges
• Conclusion

Red Hen’s Rapid Annotator provides a platform to users to annotate large chunks of data in a short span of time and with least possible efforts. It provides the features of annotating images/videos/audios/text during collaborative situation also. With Red Hen Lab’s we will try to enable users to visualize the progress of each annotator separately and annotators can notify experimenter when the annotation is finished to make the annotation work more efficient.
In Red Hen Lab’s Rapid Annotator we try to enable researchers worldwide to annotate large chunks of data in a very short period of time with least effort possible and try to get started with minimal training. I have been continuing working on Red Hen Lab's Rapid Annotator after work done by Gulshan Kumar and Vaibhav Gubta at GSoC's 20, 19 and 18.

• RA stands for Rapid Annotator
• VIA stands for Vgg Images Annotator
• PR stands for pull request
• BG stands for bug, usualy will mention an issue

• Python [Flask]
• Javascript
• HTML
• CSS

• Blog Series Link
• RA Organization Repo Link
• RA Forked Repo Link
• Multichoice feature branch
• Annotations adds-on branch

1. Going to 'Settings' of a new experiment leads to an Internal Server Error

   Description

   When I go to a new experiment and click on its settings immediately without making any uploads to this experiment (empty experiment) it leads to unexpected behavior;
   "UnboundLocalError: local variable 'pngImageB64String' referenced before assignment"
   The logs and screenshot describes it more. It gets Internal Server Error in production mode, and in debug mode it is as follows.

   Screenshot

   

   Logs

   

2. Unexpected behavior when adding or removing annotators at an experiment

   Description

   When adding/removing annotators to an experiment in a some quick way (I did that just for testing the functionality but I found that), it leads to adding the same user twice sometimes which is not logical. Is it alright or not very important to look for fixing it?

   Screenshots

   1 The behavior when using delete and add annotator functionalities.

   

   2 The delayed delete

   

3. Export results to sheet as CSV doesn't work

   Description

   Export results to sheet of an experiment works only as ".xlsx" files and "csv" doesn't work, it leads to unexpected behavior
   "UnboundLocalError: local variable 'app' referenced before assignment", screenshot attached below shows that.
   I found that due to missing import as I discovered I will submit a pull request to solve that.

   Screenshots

   

4. View large experiments and results viewing take very long time

   Description

   Currently the loading time of viewing a large experiment is very big as Dr. Peter mentioned to me and Gulshan, I have reviewed the code base and found an issue in the pagination behavior it takes processing time for the whole data, while it is expected to be done only on the portion of data which will be displayed to the user as per page request. So I will work on that to optimize that time and make lazy-loading instead of the current.

1. Fixed view settings bug of a new/empty experiment

   Description

   This pull request solves the issue of viewing settings of a new experiment

   Screenshot

   

2. Fixed adding/removing annotators bug

   Description

   This pull requests solves this issue Unexpected behavior when adding or removing annotators at an experiment #47
   It is a simple solution as I found the location reloading in unnecessary and making the unexpected behavior because immediate reloading makes the code in an inconsistent state and I didn't get what is their purpose so commenting them fixed that issue.

   Screenshots

   1. Firefox is working right now

   

   2. Making quick adding is working correctly as well as removing, everything is fine. (this on Google Chrome)

   

3. Fixed no progress alert showing when removing all annotators

   This pull request is a complementary to the latest pull request which is merged I discovered there is a problem when removing all annotators, the alert was causing some problems so I have solved it.

4. Fixed export results to csv issue

   Description

   This pull request solves Export results to sheet as CSV doesn't work #50

   Screenshot

   

5. Fixed loading time of viewing an experiment issue

   Description

   This pull request solves View large experiments and results viewing take very long time #51

   Screenshot

   Left side is after being fixed, right side is before.

   

6. Video Snippet Annotated column added to results exporting at concordance experiments

   Description

   Added a new column to the concordance experiments exporting results "Video Snippet Annotated" which is simply the same as Video Snippet but with start and end modified based on the experiment's display time [before_time and after_time]
   By applying the following equations

   new_start = video_snippet_start - exp.display_caption.before_time
   new_end = video_snippet_end + exp.display_caption.after_time

   and replacing those with the existing on the video snippet using regular expression.

   Screenshot of an experiment

   

7. View results filtered by annotation level and labels feature

   Description

   Added new two select boxes to allow viewing results filtered by annotation level and annotation level's labels.

   Screenshot

   

8. Reordering the annotation levels in a more easier way and more efficent

   Description

   Making the reordering of current existing annotation levels was a hard and need to open each annotation level to check what number is given for that and then keep all of them in your mind to make a new level or just change an existing annotation level 's level number so In this pull request proposing an easier way to do that as explained in the following screenshot.

   Screenshot

   

9. Multichoice feature branch merging to master branch

   The following features are added from multi-choice feature branch

   1. Allowing multi-choice annotation levels. i.e. (select multiple labels for same annotation level and add a field for each label as other text i.e. commenting on each selected label from the annotator)

   Multichoice annotation process and viewing results
   

   Adding a new level
   

   1. Export results at long-format (New exporting results format)
      

   2. Export results at wide-format (New exporting results format)
      

   3. Optimized the existing exporting results time

   This pull request contains database changes and new added package to apply them you need that

   # to install flask-migrate package
   pip3 install -r requirements.txt
   # Initialize the flask-migrate files.
   flask db init
   # Make a migration, it is like commit it and a comment beside.
   flask db migrate -m "Initial migration." # OR just add your comment right there  
   # Then apply your migration to the database, it is like push.
   flask db upgrade  
   

10. Continue a specific experiment using exported wide results format feature

    Description

    To allow experimenter modifying the results exported file (at wide format) and continue from the point they have edited their results to a new experiment. for example removing a row from the sheet (file)

    So the following changes has been needed to be added to wide format results export

    1. Added all attributes of file (file name, content, concordance line number; for concordance experiments and edge link and rest of them)
    2. Merged the concordance file data to the exported results wide format file (in case of it is a concordance uploaded type)

    So to continue an experiment you need to

    1. Specify which experiment to continue (select box based on your experiments)
    2. Upload the exported results in wide format
    3. Give it a new name (default: Copy of [old experiment name] [id])
    4. Give it a description (default: "")
       

    Effects are same as creating a new experiment based on the old experiment results file

    Screenshots

    1. Copy of image experiment
       

    2.Concordance experiment (it creates a new concordance file based on the given results file)
    

    3. Text experiment
    

11. New feature added and modifications on exporting results and solving view results with no annotators issue

    1. Solves the problem of viewing results of an experiment while it has no annotators

    this due to the hard coded id sent when len(annotators) = 0 at line 73 @RapidAnnotator-2.0/rapidannotator/modules/add_experiment/views.py

    Before

    

    After

    

    2. Allowing experiment's order to be modified between normal and random

    

    3. Disabled uploading once the experiment has already a done uploaded concordance file

    

    4. Exporting results of concordance at wide format will be only contains the edge link if it is new scape type

    

12. Annotations adds-on (coordinates feature)

    Description

    The following pull request adding is an adds-on to the current annotation process system to allow the experiment get more information about the data and use them with their process of information and features extraction and get more intuition about which part of the data is meant for this annotation

    Usage

    This feature meant to be used by the experimenter in the step of modeling and preprocessing.

    Screenshots

    1. Image experiments VIA Vgg images annotators, I have integrated the images part to Rapid annotator
       

    2. Audio/Video Built a time controller for selecting a time start and end of the video those controllers (buttons and ion-slider manages the video running duration and starting and ending time
       

    3. Text experiments extended the current base highlighting feature of concordance to be able to use on the normal experiment also
       

1. Fixed graph reloading at the settings of an experiment .
2. Found a new bug when exporting the results as CSV.
3. Updated the setup of adding annotation level; to allow marking on multi choice level
4. Finished the needed UI functionalities at the annotation process for supporting the new multi-choice functionality.
5. Finished the required database schema changes for multi-choice functionality.
6. Edited the annotation process endpoint to adapt with the new request structure.
7. Keeping working on it and testing and finding bugs on the new added or existing code base.

1. Optimized the viewing of an experiment loading time in terms of large experiments files I liked doing that
2. Exporting results after making the multi-choice functionality at long format
3. Edited exporting results at the concordance exp. type, added new column of video snippet annotated changed, start and end times, offsets

Filtering view results based on a specific annotation level or annotation level label

1. Added ordering the selection of label at annotation process (new column added at long-format)
2. Finished wide format at exporting results
3. Added warning when user changes the multi-choice option of an annotation level
4. Finished viewing a specific entry within the changes on multi-choice levels
5. Working on changing a specific file annotation from viewing results page
6. Working loading time of exporting an experiment results at standard format

1. Allowed changing a specific file annotation from viewing results page
2. Optimized loading time of exporting an experiment results at standard format and wide format
3. Seeing the level order number of a particular annotation level (level number attr.) while setting an experiment and being able to change that number if needed in a way more efficient than we currently have.

Re-uploading an experiment and continue to based on the results exported in wide format feature

1. Fixed the bug of going to results when there is no annotator
2. Finished disabling uploading after first uploaded done on concordance
3. Added changing order of an experiement
4. Changed display edge link for new scape only
5. Look at VIA and checked out if it could be integrated to RA or not

1. Working on switch display order functionality (figure out its effect)
2. Working on the VIA(VGG-Images-Annotations) integration to allow new annotation abilities, i.e. adding polygon boxes and starting and ending of the videos Current UI updates integrated

1. Support Highlight feature of text annotation experiment for non-concordance experiments
2. Working on adding UI components for the video and audio experiments
3. Going to edit the database to store the annotations as JSON column type

Integrated VIA and added audio/video controllers for time range from scratch as well as extended the highlighting feature and fixed a lot of bugs while working on those tasks which I call the annotations adds-on and add it into the database, I would like to explain what structure has been followed

• Image experiments it contains an array of shapes which was drawn on the image
• Audio/Video experiments a json object contains starting time and ending time in seconds
• Text contains the first index of the selected character within the highlighted text

• Always I think starting in new something is the most hard part of the story, so getting into the project for the first time was a challenging
• VIA integration and modifying to to be compatible with RA it was the hardest part (10k lines of code had to get what is going on and then start writing some code to make it works for RA was)
• The time controllers and range slider for the annotation adds-on also was a challenging part to make the video synchronized with the controllers buttons and inputs but it was interesting

• Viewing results filtration based on annotation level and label @PR07
• Reordering the annotation levels in a more easier way and more efficient @PR08
• Multi-choice selection of labels for an annotation level and added text label for each selected label @PR09
• Continue a specific experiment using its result @PR10
• Allowing experiment's order to be modified between normal and random @PR11
• Annotations adds-on and coordinates on data @PR12

• Fixed viewing setting when there are no files on the experiment @BG01 solved on @PR01
• Adding and removing annotators was getting issues on firefox specifically and had some issues generally because of it was doing reload simultaneously with sending request @BG02 solved on @PR02 and @PR03
• Exporting results as CSV file was not working correctly @BG03 solved on @PR04
• Loading time of exporting and viewing results takes a lot of time so it is optimized @BG04 solved on @PR05
• Video snippet column was added to the exported results @PR06
• Solves the problem of viewing results of an experiment while it has no annotators @PR11
• Disabled uploading once the experiment has already a done uploaded concordance file @PR11
• Exporting results of concordance at wide format will be only contains the edge link if it is new scape type @PR11