Short Answer Responses ‐ Overview

Updated: 2023-11-07

Feature Description:

Short Answer Responses are a new feature that spans play, host and networking and enables students to enter short-answer responses into a text box instead of selecting from a predefined set of multiple-choice options. Below is an excerpt from the Figma files showing the desired input condition on play:

Requirements:

While the input itself is straightforward, the implications of this feature are many. To successfully integrate this feature, we need to process the incoming data from the student in two ways. Firstly, we need to normalize the data so that all student information has the same shape. Secondly, we need to evaluate equivalence between each student answer and the correct answer: this allows us to assign points for correct answers as well as group similar incorrect answers for features like Real-Time Responses and Featured Mistakes. These two parts make up the flow of the data from the input to host. The following subsections will first outline this broad flow of data and then break down both normalization and equivalence checks in further detail.

For more detailed information on how specific equivalence should be handled, see the below document: RightOn PWA1.1 - Short Answer Responses - Answer Handling Coordination

Overall Data Flow:

The data flow for the Short Answer Responses is as follows:

play:

• Student enters answer in QuillJs pad: QuillJS delta is written to localstorage onchange of pad, the quickest way of writing and fetching data to the pad in case the user refreshes

• Student submits answer: Data is normalized using handleNormalizeAnswers() (more info on this in Normalize Answer section, below). This data is stored as a ITeamAnswerContent object, located in the ITeamAnswer object. TeamAnswerContent has the following shape:

export interface INormAnswer {
    [AnswerType.NUMBER]: number[];
    [AnswerType.STRING]: string[];
    [AnswerType.EXPRESSION]: string[];
}

export interface ITeamAnswerContent {
  delta?: string;
  rawAnswer?: string; 
  normAnswer?: INormAnswer[];
  percent?: number;
  multiChoiceAnswerIndex?: number | null;
  isSubmitted?: boolean;
  currentState: GameSessionState | null;
  currentQuestionIndex: number | null;
}

The ITeamAnswerContent object stores the delta for easy writing to the Quill object. It stores the rawAnswer to preserve the actual input of the student with mild cleanup (removal of line breaks, etc.) for use on host. normAnswer stores an array of normalized subobject, each representing a section of the answer that may be the correct answer. This is the array that will be compared to the correct answer and to other answers. This object is created via CreateTeamAnswer

• Host receives student answers: As students submit answers, host parses each of these answers, comparing their INormAnswer arrays to the correct answer and previously submitted answers. It does this via buildShortAnswerResponses() that is called via the subscription to CreateTeamAnswer. As this comparison is happening, host is then writing a IResponse object to the Question on the backend (none of the apps subscribes to this, so it doesn't trigger any other actions). This prevents a cascading of subscriptions and allows us to save all answer data to be easily retrieved via the question (either for analytics or later on in host). The shape of the IResponse is here:

export interface IResponse {
    value: string
    normAnswer: INormAnswer[];
    isCorrect: boolean
    isSelectedMistake: boolean
    count: number
    teams: Array<IResponseTeam>
}

export interface IResponseTeam {
    team: string
    id: string
    confidence: ConfidenceLevel
}

IResponse contains all normalized answers, the teams that have selected them, the number of times they've been selected. As players answer the question, this object is updated on the backend immediately so that student's answers aren't lost in the event that host crashes. The IResponse object is read on refresh for host.

The below diagram summarizes this flow:

Processing of the data in this way allows for:

1. Distribute the normalization calculations so that each app only handles one answer (instead of sending it all to host).
2. Centralize all comparisons with one data object (so that host can continually compare new answers to already vetted answers).
3. Locate shortAnswerResponses separate from choices and teamanswers so that they don't conflict as new answers come and and so that it can be used later in the app or during analytics.