Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create winnowing module for programming exercises #315

Merged
merged 91 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
505dd2e
Bump next from 13.5.6 to 14.1.1 in /playground
dependabot[bot] May 10, 2024
80cc39a
Merge branch 'develop' into dependabot/npm_and_yarn/playground/next-1…
maximiliansoelch May 11, 2024
7769e87
use unique keys across artemis instances
dmytropolityka May 12, 2024
88e5ba7
set values from .env and .ini
dmytropolityka May 13, 2024
cfc9f1f
change authorization process
dmytropolityka May 19, 2024
a376fcb
extend database to store artemis url
dmytropolityka May 19, 2024
d356749
fix bugs
dmytropolityka May 19, 2024
1757ea8
Merge branch 'develop' into feature/decouple-artemis-from-athena
dmytropolityka May 19, 2024
7df5183
fix linting issues
dmytropolityka May 19, 2024
6f837a1
add further deployments
dmytropolityka May 19, 2024
749aeae
remove dotenv
dmytropolityka May 19, 2024
cf8bd83
remove dotenv and unused code
dmytropolityka May 19, 2024
9348bf3
adjust deployment units
dmytropolityka May 19, 2024
f014359
make deployments not compulsory
dmytropolityka May 19, 2024
76df5d9
bugs
dmytropolityka May 20, 2024
bc293fa
more logging
dmytropolityka May 20, 2024
61dfd0a
more logging
dmytropolityka May 20, 2024
92b5424
fix bug
dmytropolityka May 20, 2024
2db5fa9
rename unique constraint
dmytropolityka May 20, 2024
c95634f
Update __main__.py
dmytropolityka May 27, 2024
646d51e
Merge branch 'develop' into dependabot/npm_and_yarn/playground/next-1…
maximiliansoelch May 27, 2024
1f552cc
Bump mysql2 from 3.9.7 to 3.9.8 in /playground
dependabot[bot] May 30, 2024
af34b89
rename artemis_url to lms_url
dmytropolityka Jun 2, 2024
4ae0c27
Merge remote-tracking branch 'origin/dependabot/npm_and_yarn/playgrou…
dmytropolityka Jun 2, 2024
e0be309
make playground work for multi-instance setup; add non-graded feedbac…
dmytropolityka Jun 2, 2024
8fb8fd1
Merge remote-tracking branch 'origin/dependabot/npm_and_yarn/playgrou…
dmytropolityka Jun 3, 2024
ea0696e
Merge branch 'develop' into feature/decouple-artemis-from-athena
maximiliansoelch Jun 6, 2024
c74b112
rename artemis_url into lms_url, further occurrences
dmytropolityka Jun 7, 2024
0f45254
rename artemis into lms, further occurrences
dmytropolityka Jun 7, 2024
4324ea6
Merge branch 'develop' into feature/decouple-artemis-from-athena
dmytropolityka Jun 7, 2024
7f1a317
update user and key
dmytropolityka Jun 7, 2024
2cb90bc
Revert "update user and key"
dmytropolityka Jun 9, 2024
d93daf7
remove unnecessary yarn lock file
dmytropolityka Jun 9, 2024
8fad620
Merge remote-tracking branch 'origin/feature/playground-self-learning…
dmytropolityka Jun 9, 2024
ce88507
Merge branch 'feature/decouple-artemis-from-athena' into feature/play…
dmytropolityka Jun 9, 2024
d763ed1
update experiments
dmytropolityka Jun 9, 2024
cd2474b
remove server config prior to deployment
dmytropolityka Jun 9, 2024
4a6091b
Add new module for winnowing algorithm
marlon-luca-bu May 12, 2024
1b8d631
Add AST creation in winnowing module
marlon-luca-bu Jun 16, 2024
6bb3cde
Add specific AST creation for winnowing module
marlon-luca-bu Jun 21, 2024
c997586
adapt dockerfile
dmytropolityka Jun 21, 2024
4ff298d
Revert copypaste erros
marlon-luca-bu Jun 23, 2024
6f60c8b
add debug statement
dmytropolityka Jun 24, 2024
0aeab86
modify dockerfile
dmytropolityka Jun 24, 2024
a1a050c
Add correct implementation of winnowing
marlon-luca-bu Jun 30, 2024
892057e
change signature of suggest_feedback for other modules
dmytropolityka Jul 2, 2024
17e872a
Merge remote-tracking branch 'origin/feature/playground-self-learning…
dmytropolityka Jul 2, 2024
bd97e9a
Add feedback provider
marlon-luca-bu Jul 3, 2024
01261d6
change themis to winnowing
marlon-luca-bu Jul 3, 2024
fea9e88
Debug
marlon-luca-bu Jul 4, 2024
81d0220
Fix merge conflict
marlon-luca-bu Jul 4, 2024
b28265d
add localhost to server configuration
dmytropolityka Jul 5, 2024
21b1249
modules have information whether they support non graded feedback req…
dmytropolityka Jul 7, 2024
8c2bc65
add differentiation whether to include to client code
dmytropolityka Jul 7, 2024
6229a4b
fix mypy error
dmytropolityka Jul 7, 2024
5c40f7b
Add correct way of similiarity measurement
marlon-luca-bu Jul 8, 2024
790c07c
Merge branch 'develop' into feature/playground-self-learning-feedback
FelixTJDietrich Jul 15, 2024
62b66cd
Merge branch 'develop' into feature/playground-self-learning-feedback
FelixTJDietrich Jul 15, 2024
653e8e5
Merge branch 'develop' into feature/playground-self-learning-feedback
FelixTJDietrich Jul 16, 2024
ba8d0e9
Update Dockerfile
dmytropolityka Jul 17, 2024
c0a3d14
Fix failing comparison
marlon-luca-bu Jul 18, 2024
f4e8adf
Add correct winnowing of python code
marlon-luca-bu Jul 18, 2024
058daa3
Resolve Mergeconflict
marlon-luca-bu Jul 18, 2024
b396d02
Fix typo
marlon-luca-bu Jul 18, 2024
7a134a4
poetry.lock
marlon-luca-bu Jul 18, 2024
2958c5f
ATTENTION: This commit needs to be revertedgit add .
marlon-luca-bu Jul 18, 2024
fb795cd
update default port
dmytropolityka Jul 19, 2024
059bbe8
revert merged dockerfile
dmytropolityka Jul 19, 2024
5c80031
Merge branch 'feature/playground-self-learning-feedback' into feature…
dmytropolityka Jul 19, 2024
520b212
Merge branch 'feature/create_module_programming_winnowing' of github.…
marlon-luca-bu Jul 19, 2024
8099a5d
fix issues
dmytropolityka Jul 19, 2024
e949b0f
import apted module
dmytropolityka Jul 19, 2024
436ca55
push lock files
dmytropolityka Jul 19, 2024
b765b7b
include dependency module in dockerfile
dmytropolityka Jul 19, 2024
0266448
Resolve mergeconflict
marlon-luca-bu Jul 20, 2024
b19a56c
Fix small bug
marlon-luca-bu Jul 21, 2024
bada946
Small adaption
marlon-luca-bu Jul 23, 2024
f03c4a9
Resolve mergeconflict
marlonbucciarelli Nov 15, 2024
afa55b4
Move winnowing module to new package structure
marlonbucciarelli Nov 16, 2024
869b508
Merge branch 'develop' into feature/create_module_programming_winnowing
marlonbucciarelli Nov 30, 2024
4a33051
Revert pydantic version adaption in athena module
marlonbucciarelli Nov 30, 2024
9d65cf9
Remove adpted dependencies
marlonbucciarelli Nov 30, 2024
7ac0e6f
Revert Merge conflicts changes and bugs - make module runnable again
marlonbucciarelli Nov 30, 2024
7984678
add poetry.lock again
marlonbucciarelli Nov 30, 2024
b66b807
Add ignore path to prospector
marlonbucciarelli Nov 30, 2024
3b1c49a
Add ignore path to prospector
marlonbucciarelli Nov 30, 2024
785c965
Add missing runconfigs
marlonbucciarelli Dec 1, 2024
45d03d1
Merge branch 'develop' into feature/create_module_programming_winnowing
maximiliansoelch Dec 2, 2024
39a67bf
Remove unnecessary init.py files + align pydantic version across modules
maximiliansoelch Dec 3, 2024
cb842f4
Update athena core module in winnowing and apted module
maximiliansoelch Dec 4, 2024
2aecaaf
Fix module config for apted & winnowing module
maximiliansoelch Dec 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions athena/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ httpx = "^0.24.1"
gitpython = "^3.1.41"
sqlalchemy = {extras = ["mypy"], version = "^2.0.21"}
psycopg2 = "^2.9.9"
pydantic = "1.10.13"
marlon-luca-bu marked this conversation as resolved.
Show resolved Hide resolved

[tool.poetry.group.dev.dependencies]
types-requests = "^2.31.0.8"
Expand Down
8 changes: 8 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,14 @@ services:
- postgres
image: ls1tum/athena_module_programming_apted:${ATHENA_TAG:-develop}

module_programming_winnowing:
hostname: module-programming-winnowing
env_file:
- ${ATHENA_ENV_DIR:-./env_example}/module_programming_winnowing.env
depends_on:
- postgres
image: ls1tum/athena_module_programming_winnowing:${ATHENA_TAG:-develop}

module_modeling_llm:
hostname: module-modeling-llm
env_file:
Expand Down
8 changes: 8 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,11 @@ services:
- llm_core
ports:
- "5008:5008"

module_programming_winnowing:
hostname: module-programming-winnowing
build: ./module_programming_winnowing
depends_on:
- athena
ports:
- "5009:5009"
3 changes: 3 additions & 0 deletions env_example/module_programming_winnowing.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
PRODUCTION=1
SECRET=12345abcdef
DATABASE_URL=postgresql://postgres:password@postgres:5432/athena
Empty file added modules/__init__.py
marlon-luca-bu marked this conversation as resolved.
Show resolved Hide resolved
Empty file.
Empty file added modules/programming/__init__.py
marlon-luca-bu marked this conversation as resolved.
Show resolved Hide resolved
Empty file.
4 changes: 2 additions & 2 deletions modules/programming/module_example/module_example/__main__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
Entry point for the module_example module.
Entry point for the module_example.
"""
import random
from typing import List, Any
Expand All @@ -13,7 +13,7 @@

@config_schema_provider
class Configuration(BaseModel):
"""Example configuration for the module_example module."""
"""Example configuration for the module_example."""
debug: bool = Field(False, description="Whether the module is in **debug mode**. This is an example config option.")


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ athena = { git = "https://github.com/ls1intum/Athena.git", rev = "ccd00cf8346e76
apted = "^1.0.3"
antlr4-python3-runtime = "^4.13.1"
javalang = {git = "https://github.com/c2nes/javalang"}
pydantic = "1.10.13"

[tool.poetry.group.dev.dependencies]
types-requests = "^2.31.0.8"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.venv
.vscode
__pycache__
.DS_Store
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"python.pythonPath": "./.venv/bin/python",
"python.analysis.typeCheckingMode": "basic",
}
30 changes: 30 additions & 0 deletions modules/programming/module_programming_winnowing/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# syntax=docker/dockerfile:1

# This is the Dockerfile for the module_programming_winnowing.

FROM python:3.11
LABEL org.opencontainers.image.source=https://github.com/ls1intum/Athena

# Environment variable Python in Docker
ENV PYTHONUNBUFFERED=1

WORKDIR /code

# Poetry
RUN pip install --no-cache-dir poetry==1.5.0

# Dependencies
COPY pyproject.toml poetry.lock ./
# athena module (from the Dockerfile in the athena folder)
COPY --from=athena /code /athena
COPY --from=module_programming_apted /code /module_programming_apted
marlon-luca-bu marked this conversation as resolved.
Show resolved Hide resolved
# install dependencies
RUN poetry config virtualenvs.create true \
&& poetry config virtualenvs.in-project true \
&& poetry install --no-interaction --no-ansi

# Project files
COPY . ./

# poetry scripts don't work here
CMD poetry run python -m module_*
8 changes: 8 additions & 0 deletions modules/programming/module_programming_winnowing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Start Directly
`poetry run module`

# Start with Docker
`docker-compose up --build`

# Start with Docker in Production Mode
`docker-compose up --env-file .env.production --build`
Empty file.
4 changes: 4 additions & 0 deletions modules/programming/module_programming_winnowing/module.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[module]
name = module_programming_winnowing
type = programming
port = 5009
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
"""
Entry point for the module_programming_winnowing module.
"""
import random
from typing import List, Any, cast
from pydantic import BaseModel, Field

from athena import app, config_schema_provider, submissions_consumer, submission_selector, feedback_consumer, feedback_provider, evaluation_provider, emit_meta
from athena.programming import Exercise, Submission, Feedback, get_stored_feedback_suggestions, \
count_stored_submissions, get_stored_submissions
from athena.logger import logger
from athena.storage import store_exercise, store_submissions, store_feedback, store_feedback_suggestions
from module_programming_winnowing.convert_code_to_ast.get_feedback_methods import get_feedback_method
from module_programming_winnowing.feedback_suggestions.feedback_suggestions import create_feedback_suggestions
from module_programming_winnowing.feedback_suggestions.remove_overlapping import filter_overlapping_suggestions
from module_programming_winnowing.feedback_suggestions.remove_suspicious import filter_suspicious


@config_schema_provider
class Configuration(BaseModel):
"""Example configuration for the module_programming_winnowing module."""
debug: bool = Field(False, description="Whether the module is in **debug mode**. This is an example config option.")


@submissions_consumer
def receive_submissions(exercise: Exercise, submissions: List[Submission], module_config: Configuration):
logger.info("receive_submissions: Received %d submissions for exercise %d", len(submissions), exercise.id)
for submission in submissions:
logger.info("- Submission %d", submission.id)
zip_content = submission.get_zip()
# list the files in the zip
for file in zip_content.namelist():
logger.info(" - %s", file)
# Do something with the submissions
logger.info("Doing stuff")

# Example use module config
# If you are not using module_config for your module, you can remove it from the function signature
logger.info("Config: %s", module_config)
if module_config.debug:
emit_meta('debug', True)
emit_meta('comment', 'You can add any metadata you want here')

# Add data to exercise
exercise.meta["some_data"] = "some_value"
logger.info("- Exercise meta: %s", exercise.meta)

# Add data to submission
for submission in submissions:
submission.meta["some_data"] = "some_value"
logger.info("- Submission %d meta: %s", submission.id, submission.meta)

store_exercise(exercise)
store_submissions(submissions)


@submission_selector
def select_submission(exercise: Exercise, submissions: List[Submission]) -> Submission:
logger.info("select_submission: Received %d submissions for exercise %d", len(submissions), exercise.id)
for submission in submissions:
logger.info("- Submission %d", submission.id)
# Do something with the submissions and return the one that should be assessed next
return submissions[0]


@feedback_consumer
def process_incoming_feedback(exercise: Exercise, submission: Submission, feedbacks: List[Feedback]):
logger.info("process_feedback: Received %d feedbacks for submission %d of exercise %d", len(feedbacks),
submission.id, exercise.id)
logger.info("process_feedback: Feedbacks: %s", feedbacks)

programming_language = exercise.programming_language.lower()
# Currently only works with Java and Python - can be extended with more languages if the grammar is available
if programming_language not in ["java", "python"]:
logger.info("The winnowing module currently only works with Java and Python. Not consuming feedback.")
return

# Remove unreferenced feedbacks
feedbacks = list(filter(lambda f: f.file_path is not None and f.line_start is not None, feedbacks))

# Add method metadata to feedbacks
feedbacks_with_method = []
for feedback in feedbacks:
feedback_method = get_feedback_method(submission, feedback, programming_language)
if feedback_method is None:
# don't consider feedback without a method
continue
logger.debug("Feedback #%d: Found method %s", feedback.id, feedback_method.name)
feedback.meta["method_name"] = feedback_method.name
feedback.meta["method_code"] = feedback_method.source_code
feedback.meta["method_line_start"] = feedback_method.line_start
feedback.meta["method_line_end"] = feedback_method.line_end
feedback.meta["method_ast"] = feedback_method.ast
feedbacks_with_method.append(feedback)
feedbacks = feedbacks_with_method

# find all submissions for this exercise
exercise_submissions = cast(List[Submission], list(get_stored_submissions(exercise.id)))

# create feedback suggestions
logger.info("Creating feedback suggestions for %d feedbacks", len(feedbacks))
feedback_suggestions = create_feedback_suggestions(exercise_submissions, feedbacks, programming_language)

# additionally, store metadata about how impactful each feedback was, i.e. how many suggestions were given based on it
for feedback in feedbacks:
# count how many suggestions were given based on this feedback
feedback.meta["n_feedback_suggestions"] = len(
[f for f in feedback_suggestions if f.meta["original_feedback_id"] == feedback.id])
# store the information on the suggestions as well for quicker access later
for suggestion in feedback_suggestions:
if suggestion.meta["original_feedback_id"] == feedback.id:
suggestion.meta["n_feedback_suggestions"] = feedback.meta["n_feedback_suggestions"]

# save to database
# type: ignore
store_feedback_suggestions(feedback_suggestions)
for feedback in feedbacks:
store_feedback(feedback)

logger.debug("Feedbacks processed")

@feedback_provider
def suggest_feedback(exercise: Exercise, submission: Submission, is_graded: bool, module_config: Configuration) -> List[Feedback]:
logger.info("suggest_feedback: Suggestions for submission %d of exercise %d were requested", submission.id,
exercise.id)
# Do something with the submission and return a list of feedback
# ThemisML currently only works with Java
if exercise.programming_language.lower() not in ["java", "python"]:
logger.info("The Winnowing module currently only works with Java and Python. Returning no suggestions.")
return []

suggested_feedbacks = cast(List[Feedback], list(get_stored_feedback_suggestions(exercise.id, submission.id)))
logger.debug("Found %d feedback suggestions (unfiltered)", len(suggested_feedbacks))
suggested_feedbacks = filter_suspicious(suggested_feedbacks, count_stored_submissions(exercise.id))
logger.debug("Found %d feedback suggestions (removed suspicious suggestions)", len(suggested_feedbacks))
suggested_feedbacks = filter_overlapping_suggestions(suggested_feedbacks)
logger.debug("Found %d feedback suggestions (removed overlapping suggestions)", len(suggested_feedbacks))

logger.info("Suggesting %d filtered feedback suggestions", len(suggested_feedbacks))
logger.debug("Suggested Feedback suggestions: %s", suggested_feedbacks)

return suggested_feedbacks


# Only if it makes sense for a module (Optional)
@evaluation_provider
def evaluate_feedback(exercise: Exercise, submission: Submission, true_feedbacks: List[Feedback], predicted_feedbacks: List[Feedback]) -> Any:
logger.info(
"evaluate_feedback: Evaluation for submission %d of exercise %d was requested with %d true and %d predicted feedbacks",
submission.id, exercise.id, len(true_feedbacks), len(predicted_feedbacks)
)

# Do something with the true and predicted feedback and return the evaluation result
# Generate some example evaluation result
evaluation_results = []
true_feedback_embeddings = [random.random() for _ in true_feedbacks]
predicted_feedback_embeddings = [random.random() for _ in predicted_feedbacks]
for feedback, embedding in zip(predicted_feedbacks, predicted_feedback_embeddings):
feedback_evaluation = {
"feedback_id": feedback.id,
"embedding": embedding,
"has_match": len([t for t in true_feedback_embeddings if abs(t - embedding) < 0.1]) > 0,
"correctness": random.random()
}
evaluation_results.append(feedback_evaluation)

return evaluation_results


if __name__ == "__main__":
app.start()
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
from antlr4 import CommonTokenStream, InputStream
from antlr4.tree.Tree import ParseTreeWalker
from module_programming_apted.convert_code_to_ast.languages.python.Python3Lexer import Python3Lexer
marlon-luca-bu marked this conversation as resolved.
Show resolved Hide resolved
from module_programming_apted.convert_code_to_ast.languages.python.Python3Parser import Python3Parser
from module_programming_apted.convert_code_to_ast.languages.java.JavaLexer import JavaLexer
from module_programming_apted.convert_code_to_ast.languages.java.JavaParser import JavaParser
from module_programming_apted.convert_code_to_ast.languages.python.Python3MethodParserListener import \
MethodParserListener as PythonMethodParserListener
from module_programming_apted.convert_code_to_ast.languages.java.JavaMethodParserListener import \
MethodParserListener as JavaMethodParserListener

# TODO: DO I need the to_ast method?

# Grammars for programming languages have different parse rules
JAVA_PARSE_RULE = "compilationUnit"
PYTHON_PARSE_RULE = "file_input"

def parse_java_file(source_code: str):
return parse_file(source_code, JavaLexer, JavaParser, JAVA_PARSE_RULE, JavaMethodParserListener)


def parse_python_file(source_code: str):
return parse_file(source_code, Python3Lexer, Python3Parser, PYTHON_PARSE_RULE, PythonMethodParserListener)


def parse_file(source_code, lexer_class, parser_class, parse_rule, listener_class):
input_stream = InputStream(source_code)
lexer = lexer_class(input_stream)
stream = CommonTokenStream(lexer)
parser = parser_class(stream)
tree = getattr(parser, parse_rule)()

listener = listener_class(parser)
walker = ParseTreeWalker()
walker.walk(listener, tree)
print(listener.methods)

return listener.methods.copy()


def parse(source_code: str, programming_language: str):
if programming_language == "java":
return parse_java_file(source_code)
if programming_language == "python":
return parse_python_file(source_code)
raise ValueError(f"Unsupported programming language: {programming_language}")


if __name__ == "__main__":
# file_path2 = "../test_files/test_java_1.java"
# parse_java_file(file_path2)

code = """def process_numbers(numbers):
total = 0
for number in numbers:
if number % 2 == 1:
total += number
else:
total -= number
if total > 0:
print("Positive total:", total)
else:
print("Non-positive total:", total)"""
code1 = parse_python_file(code)
code2 = parse_python_file(code)
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
from typing import Optional

from athena.programming import Submission, Feedback
from module_programming_apted.convert_code_to_ast.extract_method_and_ast import parse
from module_programming_apted.convert_code_to_ast.method_node import MethodNode
from athena.logger import logger


def get_feedback_method(submission: Submission, feedback: Feedback, programming_language: str) -> Optional[MethodNode]:
"""Find method that the feedback is on"""
if feedback.file_path is None or feedback.line_start is None:
return None
try:
code = submission.get_code(feedback.file_path)
except UnicodeDecodeError:
logger.warning("File %s in submission %d is not UTF-8 encoded.", feedback.file_path, submission.id)
return None
methods = parse(code, programming_language)
for m in methods:
if m.line_start is None or m.line_end is None:
continue
# method has to contain all feedback lines
if m.line_start <= feedback.line_start:
feedback_line_end = feedback.line_end if feedback.line_end is not None else feedback.line_start
if m.line_end >= feedback_line_end:
return m
return None
Loading
Loading