Short Answer Response ‐ Equality Checks

Updated: 2023-11-07

Per the Overview, Short Answer Response data is processed first via normalization in play and then via equality checks with both the correct and other student answers in host. This article breaks down the logic and functions of those equality checks but for more information on the normalization, please see Editing Short Answer Responses ‐ Answer Normalization.

Picking up from that normalization, that normalized data is written to the GameSession TeamAnswer via CreateTeamAnswer when a user submits their short answer response. host has a subscription to CreateTeamAnswer and thus will be able to receive the update TeamAnswerContent subobject via this subscription response. As students answers the question, the host app will then be receiving a series of normalized answer objects via the CreateTeamAnswer subscription. host's job in this context is to determine which of these normalized answers matches the correct answer and which of these normalized answers match previously submitted answers (on the same question).

To achieve this, we will again break down the responses first by their type and then by their specific content, allowing us to immediately discard a subset of answers that don't correspond to type. We organized the answer types in play by how reliable we think those determinations are and by how easy the equality check is. For example, we first test if an answer is numeric and then by an expression because each of these types of equality tests are easier than dealing with a more messy string. In the below algorithm, the string represents the least reliable/rigourous test so we are working on the assumption here that by the time we get to those kinds of checks, we've already extracted and compared all better types of answers (numbers and expressions), found that they didn't matched and are essentially trying some final checks for equality.

Algorithm

From src/lib/HelperFunctions.jsx in host:

export const determineAnswerType = (answer) => {
  // check if answer is numeric
  if (
    (typeof(answer) === 'number' || 
    typeof(answer) === "string") && 
    answer.trim() !== ''
    && !isNaN(answer)
  ) 
    return AnswerType.NUMBER; 
  // check if answer is an expression
  try {
    // if the answer contains mathematical operators and mathjs can parse the answer, it's an expression
    const expRegex = /[+\-*/^().]/;
    if (expRegex.test(answer) && parse(answer))
      return AnswerType.EXPRESSION;
    return AnswerType.STRING;
  } catch {
    // if the parse fails, the answer is a string
    return AnswerType.STRING;
  }
};

export const checkExpressionEquality = (normAnswer,  prevAnswer) => {
  let isSymbolicallyEqual = false;
  for (let normItem of normAnswer) {
    for (let prevItem of prevAnswer) {
      try {
        if (symbolicEqual(normItem, prevItem) || normItem.toString() === prevItem.toString()) {
          isSymbolicallyEqual = true;
          break; // Break out of the inner loop
        }
      } catch (e) {
        console.error(e);
      }
    }
    if (isSymbolicallyEqual) {
      break; // Break out of the outer loop if a match has been found
    }
  }
  return isSymbolicallyEqual;
}

export const checkEqualityWithOtherAnswers = (rawAnswer, normAnswerType, normAnswer,  prevAnswer, correctAnswerRegex) => {
  // convert to set to optimize the equality check between two arrays
  // this prevents having to use two nested for/foreach loops etc
  const prevAnswerSet = new Set(prevAnswer);
  // expression equality requires a bit more work, as we have to use mathjs to check for symbolic equality
  if (Number(normAnswerType) === AnswerType.EXPRESSION) {
    return checkExpressionEquality(normAnswer, prevAnswerSet);
  }
  return ( normAnswer.some(item => prevAnswerSet.has(item)) 
    || (Number(normAnswerType) === AnswerType.STRING && correctAnswerRegex.test(normAnswer))
    || (rawAnswer === prevAnswer.value) 
  );
}


export const buildShortAnswerResponses = (prevShortAnswer, choices, newAnswer, newAnswerTeamName, teamId) => {
  let correctAnswer = choices.find(choice => choice.isAnswer).text;
  const correctAnswerRegEx = new RegExp(`\\b${correctAnswer.toLowerCase()}\\b`, 'g');
  if (prevShortAnswer.length === 0) {
    const correctAnswerType = determineAnswerType(correctAnswer);
    if (correctAnswerType === AnswerType.NUMBER) {
      correctAnswer = Number(correctAnswer);
    } else {
      correctAnswer = correctAnswer.toLowerCase();
    }
   
    prevShortAnswer.push({
      value: correctAnswer,
      isCorrect: true,
      isSelectedMistake: false,
      normAnswer: {
        [correctAnswerType]: [correctAnswer]
      },
      count: 0,
      teams: [],
    });
  }
  const rawAnswer = newAnswer.answerContent.rawAnswer;
  let isExistingAnswer = false;

  // for each answer type in the newly submitted answer
  Object.entries(newAnswer.answerContent.normAnswer).forEach(([key, value])=>{
      // for each answer in the previous short answer array 
      prevShortAnswer.forEach((prevAnswer) => {
        // check equality based on the type of answer
        if (isExistingAnswer === false 
          && prevAnswer.normAnswer[key] 
          && checkEqualityWithOtherAnswers(rawAnswer, key, value, prevAnswer.normAnswer[key], correctAnswerRegEx)) {
          isExistingAnswer = true;
          prevAnswer.count += 1;
          prevAnswer.teams.push({name: newAnswerTeamName, id: teamId, confidence: newAnswer.confidenceLevel});
        }
    });
  });

  if (!isExistingAnswer){
    prevShortAnswer.push({
      value: rawAnswer,
      normAnswer: newAnswer.answerContent.normAnswer,
      isCorrect: false,
      isSelectedMistake: false,
      count: 1,
      teams: [{name: newAnswerTeamName, id: teamId, confidence: newAnswer.confidenceLevel}]
    });
  }
  return prevShortAnswer;
};

The equality checking begins with buildShortAnswerResponses(). The goal of this function is to generate the IResponse object that we will store in the IQuestion object in DynamoDb. This object will be a complete listing of all normalized answers, with the count of how many teams selected them and which particular teams did. This object will be used to populate the Victory charts, the Featured Mistakes feature and the Phase 2 answer options, as well as any analytics/dashboards in the future.

IResponse will always contain the correct answer first. We insert this correct answer into the Iresponse object and determine it's type using determineAnswerType(). We only have to determine the correct answer type here, as all other answers will come from the backend with their answer type already determined by play.

We then begin the equality checking process. As mentioned above, we first parse the comparisons by answer type, skipping the answers that don't have matching types. We check answer types based on ease/reliability of comparison first, beginning with numeric answers, than evaluating for expressions and finally strings. If we get a string answer type, we use includes to try to get the widest possible result. If we get an expression, we use mathjs to symbolicEqual both expressions and compare the result for equality. Finally, if they are numbers we just do a numeric javascript comparison.

Finally, there are some edge cases that are captured after this search (and are only run if no equality has been found thus far). If either side of the comparison is an expression, we try a simple string comparison. This catches scenarios where a student has entered a response in the forumla box that a teacher might not have (for example, they enter 5% as a formula and a teacher enters it as a string). This would normally get thrown out by the type comparison, so we catch it here.

Additionally, we do a last-ditch direct comparison as a final effort to try and find equality.