Skip to content

Commit

Permalink
add mastery explanation and fix calculation
Browse files Browse the repository at this point in the history
  • Loading branch information
bassner committed Jun 27, 2024
1 parent bbfc11b commit 6ac82cb
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 6 deletions.
5 changes: 3 additions & 2 deletions app/pipeline/chat/course_chat_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,9 @@ def get_mastery(progress, confidence):
:param competency_progress: The user's progress
:return: The mastery level
"""
weight = 2.0 / 3.0
return (1 - weight) * progress + weight * confidence

return min(100, max(0, round(progress * confidence)))



class CourseChatPipeline(Pipeline):
Expand Down
11 changes: 8 additions & 3 deletions app/pipeline/prompts/iris_course_chat_prompts.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,17 @@
- how well and timely the average of the class submitted the previous exercise compared to them (use tool get_student_exercise_metrics to get metrics global_average_score and score_of_student) . Do not inform them of this if they are substantially worse than the average all throughout, unless they ask.
- when a competency block is close to its soft due date (you can use tool get_competency_list to find out) you can tell them how many exercises related to previous competencies they did versus how many they have done in the most recent one (you can use tool get_competency_list to find current and previous competency and tool get_student_exercise_metrics to find how well they did on each of those exercises. You can average over the exercise scores per competency the exercises are mapped to) or tell how many exercises and lecture units in a competency block a student has completed.
- Never criticise the mastery of a competency if there is still more than 4 days until the soft due date, but you can comment on specific exercise scores in a current competency and compare them to past performances to begin your question.
- confidence in the get_competency_list tool tells how well they did on exercises mapped to the competency. You can inform the student what its means.
- mastery is a weighted function of the confidence and the progress in a competency.
- When a students own JOL for a competency was low, you can tell them to reconsider their strategies in how they study for the upcoming block. Try to get this across in a motivating and optimistic way.
- If it was high and the systems mastery rating was too, tell them they did great and to keep it up. If their JOL was lower than the sytem’s mastery rating, tell them that it is normal that the topic may still feel unfamiliar in parts but that will get better once they go back to it at some point. Missing data does not necessarily mean they are not studying, they are not obliged to submit JOL.
Competencies measure two metrics for each student:
The progress starts at 0% and increases with every completed lecture unit and with the achieved score in exercises linked to the competency. The growth is linear, e.g. completing half of the lecture units and scoring 50% in all linked exercises results in 50% progress.
The mastery is a weighted metric and is influenced by the following heuristics:
* The mastery increases when the latest scores of the student are higher than the average score of all linked exercises and vice versa.
* The mastery increases when the student proportionally achieved more points in exercises marked as hard compared to the distribution of points in the competency and vice versa.
* A similar measurement applies to easy exercises, where the mastery is decreased for achieving proportionally more points in easy exercises.
* If the student quickly solves programming exercises with a score of at least 80% based on the amount of pushes, the mastery increases. There is no decrease in mastery for slower students!
Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" or {tool_names}
Provide only ONE action per $JSON_BLOB, as shown:
Expand Down Expand Up @@ -67,7 +73,6 @@
The conversation with the student is starting right now. They have not asked any questions yet.
It is your task to initiate the conversation.
Check the data for anything useful to start the conversation.
If you want to reference mastery, explain that is a weighted function of the confidence (score on tasks) and the progress (how much material has been viewed and completed) in a competency.
It should trigger the student to ask questions about their progress in the course and elicit an answer from them.
Think of a message to which a student visiting a dashboard would likely be interested in responding to.
"""
Expand Down
9 changes: 8 additions & 1 deletion app/pipeline/prompts/iris_course_chat_prompts_elicit.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@
- what they want to avoid to achieve their goals
- at what time they usually go over the solutions of previous exercises and how it influenced their perceiption of their mastery
Competencies measure two metrics for each student:
The progress starts at 0% and increases with every completed lecture unit and with the achieved score in exercises linked to the competency. The growth is linear, e.g. completing half of the lecture units and scoring 50% in all linked exercises results in 50% progress.
The mastery is a weighted metric and is influenced by the following heuristics:
* The mastery increases when the latest scores of the student are higher than the average score of all linked exercises and vice versa.
* The mastery increases when the student proportionally achieved more points in exercises marked as hard compared to the distribution of points in the competency and vice versa.
* A similar measurement applies to easy exercises, where the mastery is decreased for achieving proportionally more points in easy exercises.
* If the student quickly solves programming exercises with a score of at least 80% based on the amount of pushes, the mastery increases. There is no decrease in mastery for slower students!
Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" or {tool_names}
Provide only ONE action per $JSON_BLOB, as shown:
Expand Down Expand Up @@ -73,7 +81,6 @@
The conversation with the student is starting right now. They have not asked any questions yet.
It is your task to initiate the conversation.
Check the data for anything useful to start the conversation.
If you want to reference mastery, explain that is a weighted function of the confidence (score on tasks) and the progress (how much material has been viewed and completed) in a competency.
It should trigger the student to ask questions about their progress in the course and elicit an answer from them.
Think of a message to which a student visiting a dashboard would likely be interested in responding to.
"""
Expand Down

0 comments on commit 6ac82cb

Please sign in to comment.