Preliminary AI Feedback for all Apollon UML Diagrams

[LeonWehrhahn]

Motivation and Context

The initial version of the module_modeling_llm was designed to primarily handle BPMN Diagrams. This enhanced implementation uses a new way to parse UML diagrams with a more generic approach making it work with all Apollon supported diagram types. The module has also been updated to utilize Langchain 0.2.15, which allows for the use of native structured output with json_mode for the latest OpenAI models. Additionally, it now supports both graded and non-graded feedback, ensuring that exercise solutions are not revealed when a student requests feedback.

The module structure is beeing refactoring in a separate PR to enhance maintainability. Including decomposing the generate_suggestions function into smaller, more cohesive units and moving the models into separate files

Description

1. The new Apollon UML parser converts Apollon JSON into a text representation similar to Mermaid. This representation retains only the most essential attributes of a UML diagram, such as elements, the relationships between them, and the properties/methods associated with these elements and relationships.

2. Since the module was originally built using an older version of Langchain, the code interfacing with Langchain has been rewritten to leverage the new LCE framework. This update also allows for the use of native structured output, eliminating the need to rely on function calling to obtain valid JSON.

3. The module now also differentiates between graded and non-graded feedback. The is_graded parameter is set to true on the Artemis side when a tutor requests feedback suggestions. When a student requests feedback, is_graded is set to false. If is_graded is false, the feedback output is passed through a filter which rewrites the title and descriptions to avoid giving away the solution.

   Example: Here we send a student's diagram submission where the start event of a BPMN diagram is missing:

   is_graded = true:
   
   is_graded = false:
   

Steps for Testing

Option 1: API Testing:

• Make direct API calls to the /feedback_suggestions endpoint of the module.
• Test with various diagram types (BPMN, UML Class, UML Activity, etc.) to ensure the new generic approach works for all Apollon-supported diagrams.
• For each diagram type, test both is_graded = true and is_graded = false scenarios.

Option 2. Athena Playground Testing:

• Access the Athena Playground.
• Use the provided examples to send requests to the Athena /feedback_suggestions endpoint.
• Verify that that appropriate Feedback is returned.

3. Graded vs. Non-Graded Testing:
   • For is_graded = true:
     • Verify that the feedback provides detailed information including specific solutions.
     • Check that all grading instructions were regarded in the feedback.
   • For is_graded = false:
     • Confirm that the feedback does not reveal the solution directly.
     • Ensure that the feedback is helpful but general, guiding the student without giving away answers.

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.

Screenshots

[FelixTJDietrich]

Please add example data for the diagrams, I could only test BPMN

[github-actions]

⚠️ Unable to deploy to test servers ⚠️

The docker build needs to run through before deploying.

[FelixTJDietrich]

LGTM