From d8fc691c1d223c8746eda7c2cbe76597bc277cf9 Mon Sep 17 00:00:00 2001 From: Pradeep Senthil Date: Tue, 26 Mar 2024 00:01:42 -0500 Subject: [PATCH] Updated instruction.js (#100) Co-authored-by: Zimu Li <131373276+neuroZ68@users.noreply.github.com> --- src/ui_elements/Components/instructions.js | 177 ++++++++++++++++----- 1 file changed, 138 insertions(+), 39 deletions(-) diff --git a/src/ui_elements/Components/instructions.js b/src/ui_elements/Components/instructions.js index aa104cb..41ed96f 100755 --- a/src/ui_elements/Components/instructions.js +++ b/src/ui_elements/Components/instructions.js @@ -3,46 +3,145 @@ import React from "react"; export default function Instructions(){ return (
-

Keybinds:

+

Instructions for Using the AVAT Platform

+

The AVAT platform is designed to be accessible for users across various academic disciplines, providing a robust interface for efficient video annotation. This document outlines the key features of the platform, including user interface details, key bindings, and the process for uploading media and annotating videos.

+

For a system that spans multiple academic disciplines, ensuring a robust interface allows a person of any experience to efficiently annotate videos. A potential user should be able to identify animals of interest easily and have a method to track them across frames in an easy and efficient manner. For a more efficient workflow, a series of key bindings was implemented to expedite each operation within the platform, therefore allowing unexperienced users to familiarize themselves with a predetermined workflow efficiently.

+

Overview

+

AVAT aims to simplify the video annotation process, allowing users to easily identify and track animals of interest across frames. A significant challenge addressed was conditioning users to adopt a predetermined workflow for increased efficiency, even without prior intensive training on our system.

+

Browser Support

+

The following browsers are supported by the AVAT platform. If you encounter any errors, we recommend using the latest version of Chrome or Firefox, as these have been extensively tested:

+ +

Direct Access (Recommended)

+

Access AVAT via https://aifarms.github.io/AVAT/.

+

Local Deployment (Frontend)

+

These instructions will help you set up a local copy of the project for development and testing purposes.

+

Prerequisites

+

You will need to have npm, firefox/chrome, and git installed on your system.

+

Installing and Building

+
    +
  1. Clone the repository to your desired location: + ```bash + git clone https://github.com/AIFARMS/AVAT
    2. +
  2. Install the required node_modules: + ```bash + npm install
    3. +
  3. Start the project locally. The website will be accessible at http://localhost:4000/. This command starts the front-end portion of the dashboard: + ```bash + npm start
    4. +
  4. To simply access the website, navigate to the build folder and open index.html.
    5. +
+

Key Bindings

+

The platform has a series of built in key binds that allow the user to employ different sampling strategies. Skipping through frames is accomplished with “e” for forward and “q” for backward, while “w” toggles the pause/play function. The “c” key is particularly useful for copying the previous annotation, thereby reducing repetitive tasks when annotating sequences with minimal changes. The “f” key is used to toggle the scrub mode, which is activated by default to allow for quick navigation through the video timeline.

+ +

User Interface

+

Video Playback and Annotation Area

+ + + +

Media Upload and Initialization

+

Uploading Videos

+
    +
  1. Access the “Upload” tab to start.
    2. +
  2. Select the media type to be annotated from the Format dropdown.
    3. +
  3. Specify the number of video streams in the “Stream Number” field.
    4. +
  4. Click “Browse” next to “Video Upload” and choose the file from the local directory.
    5. +
  5. Pre-defined ethogram labels or existing annotations can be uploaded using the “Column Upload” and “Annotation Upload” buttons.
    6. +
  6. Ensure the “Frame Rate” matches the video’s frame rate.
    7. +
  7. Set the “Skip Value” to determine the frame sampling interval.
    8. +
  8. Click “Upload” to process and display the first frame, indicating readiness for annotation.
    9. +
+

Uploading Images

+ +

Initialization workflow

+ +

Annotation Types and Process

+ +

Customizing the Ethogram with data_column.json

+

To tailor the AVAT platform's annotation system to your study's specific needs, you'll need to modify the data_column.json template. This file dictates the structure of the annotation table and the options available in dropdown menus for annotating videos. Follow the steps below to customize this file.

+

Step 1: Accessing the Template

+

The template can be found at the AVAT GitHub repository: data_column.json. Download or view this file to begin customization.

+

Step 2: Understanding the File Structure

+

The data_column.json consists of two primary sections:

+ +

Step 3: Editing the File

+

Annotation Workflow

+
    +

  1. Preparation

    +
      +
    • Before starting, ensure that your media files are ready for upload. This includes having videos in supported formats and images, if applicable, prepared in a 16:9 aspect ratio .mp4 to ensure maximum compatibility with the AVAT platform.
      • +
    +
    2. +

  2. Uploading Media

    +
      +
    • Navigate to the “Upload” tab on the platform's interface.
      • +
    • For videos, select the type of media to be annotated from the Format dropdown menu. Specify the number of video streams for upload in the “Stream Number” field.
      • +
    • Click “Browse” next to “Video Upload” and select the file from your local directory. If you have predefined ethogram labels or existing annotations, use the “Column Upload” and “Annotation Upload” buttons to upload these files.
      • +
    • Ensure the “Frame Rate” field matches the video’s frame rate for synchronization. Adjust the “Skip Value” to set the frame sampling interval according to your needs.
      • +
    • Click “Upload” to initiate the processing of the videos or images. Once complete, the first frame of the video will be displayed in the playback area, indicating readiness for annotation.
      • +
    +
    3. +

  3. Starting the Annotation Process

    +
      +
    • Begin annotating the video by using the key binds to navigate through frames and manage annotations. You can switch between different annotation modes (e.g., behavior annotation, bounding box) as needed to accurately label the video content.
      • +
    • For behavior annotations, use the 1 and a keys to add labels for behaviors or postures that don't have visual attributes.
      • +
    • To create bounding boxes around objects of interest, switch to the bounding box mode with the 2 key and use the a key to add a new bounding box. Resize and position the box using your mouse, ensuring it accurately encompasses the object.
      • +
    +
    4. +

  4. Reviewing and Adjusting Annotations

    +
      +
    • Periodically review your annotations to ensure accuracy and completeness. Use the navigation controls and key bindings to revisit previous frames and adjust annotations as necessary.
      • +
    • If you need to remove or edit an annotation, utilize the corresponding key bindings (r for removal, e and q for navigation) and mouse actions to make precise adjustments.
      • +
    +
    5. +

  5. Exporting Annotated Data

    +
      +
    • Once you have completed the annotation process, you can export your annotated data for further analysis or sharing with your team. Use the "Export" option in the navigation bar to save your work in the desired format.
      • +
    +
    6. +
+

Sampling rate

+

Ensure video frame rates are accurately inputted for synchronization. Adjust sampling values as needed for your project's requirements. + Given that manually labeling every image or frame of a video is an inefficient process, our system adopts an approach that requires users to annotate a sparse set of frames, also referred to as the sampling rate. The sampling rate is dependent on the research question, and we recommend two sampling rate methods: static sampling and dynamic sampling.

+

Static sampling is a simple strategy that aims to annotate animals on a static interval where the N intermediate frames are skipped. This strategy is perfect for videos where an set of ethogram labels has already been specified as the user is not looking out for odd or new behaviors that might occur. + For videos that have a lot of movement, we set N to be low, whereas, for videos that have small amounts of movement, we can set N to be higher. As such, the primary goal of the user is to minimize N such that key elements of the video are not left out by the skipped frames. Having N too low will result in the user wasting time annotating frames that could easily be interpolated and having a high N will result in key frames being skipped causing the user to comb through the individual frames manually, wasting time. + Assuming a suitable ethogram is provided, one can determine the N as the minimum time t it takes for a single event to happen between frames. Specifically, N would be set relative to the activity level of the animals in the video. For example, if the animals were extremely active and engaging in lots of social behaviors through the course of the video, N would be set low as it helps capture nuanced behaviors that would be missed with a high N. Likewise, if the animals were docile and not social, N would set high since not much activity would have happened between frames and the ethogram would only track behaviors that occur through the course of several frames. + Functionally, static sampling involves asking the user for N. This value should be set at the start and should be manually set by the user at all times. We recommend a default value of 1 frame per second of video as a baseline N value.

+

Dynamic sampling is a strategy that aims to annotate animals for use in the creation of an ethogram. This strategy allows the user to decide when to annotate based on the happenings in the video. This allows the collection of new behaviors and adds the ability for the user to add notes about what is occurring for help in ethogram development. Since the choice of which frames to annotate is on the user, it adds variance in the results and time were taken to complete annotations. For a trained annotator, using dynamic sampling is beneficial since it allows for frames that are not of interest to be skipped in favor of more data-rich frames. However, this sampling strategy requires a seasoned annotator. Unskilled annotators are likely to be less efficient while employing a dynamic sampling strategy since they are not aware of which frames contain the most important features. We recommend using this strategy only if the user is a trained annotator. Functionally, dynamic sampling is implemented by a pause/play option and the ability to scrub through the video. Adding functionality such as playback speed significantly helps the user as they are able to skip through the irrelevant frames with ease. Allowing the user to control the frame level is key as some frames might have changes in between single frames. + The platform has a series of built in keybinds that allow the user to employ different sampling strategies. Skipping through frames is accomplished with “e” for forward and “q” for backward, while “w” toggles the pause/play function. The “c” key is particularly useful for copying the previous annotation, thereby reducing repetitive tasks when annotating sequences with minimal changes. The “f” key is used to toggle the scrub mode, which is activated by default to allow for quick navigation through the video timeline.

+

Troubleshooting and Feedback

+

For any issues or feedback, please use the "Report" button in the top toolbar to submit a detailed report.

- There are preset keybinds setup to make it easier to use the tool without having to click around. If done properly a combination of mouse and keyboard actions speed up the annotation process. -



- * 1 : Mode Switch: Behavior

- * 2 : Mode Switch: Bounding Box

- * 3 : Mode Switch: Segmentation (Currently Disabled)

- * a : Add annotation

- * e : Skip forward frame(s)

- * q : Skip backward frame(s)

- * w : Pause/Play

- -



-

Uploading Video:

- - Ensure that you know the framerate of the video chosen to be annotated. These values should be entered into the settings tab into their "frame rate" fields. -



- Click on the right side browse button. All other buttons on the screen shuold be disabled until the video has been uploaded. Currenntly ```.mp4``` format is the best choice and tested for. Other file types such as .avi are supported but might have unintended bugs. -

- Please note that videos SHOULD be in 16:9 aspect ratio for best results. Extraneous behavior may occur if resolution is different. -



- -

Uploading Image:

- Select all of the image files needed for the various inputs. To select a group of images, they should be in the same folder. -

- Please note that images SHOULD be in 16:9 aspect ratio for best results. Extraneous behavior may occur if resolution is different. -



-

Annotations

-

- There are currently two kinds of annotations. -



-

Behavior Annotation:

- - This is for having annotations that do not have any visual attribute attached to it. This annotation is also attached by default when the bounding box or segmentation annotation is created. - -



-

Bounding Box:

- - This forms a square around the desired object. There should be small squares at the edges of the bounding box which can be used to resize the box. The number on the top left of the box is used to identify the placement of the box in the table to the right. -



) } \ No newline at end of file