Langchain #216
Langchain #216
Conversation
…into langchain Merging evaluation pipeline together
…langchain Updating the LLM setup, as well as new tests and protocols
…langchain Conflicts: pyproject.toml
batwood-1
requested review from
fabiocat93
and removed request for
fabiocat93
There was a problem hiding this comment.
hi @batwood-1 , i have left some comments. I feel some changes are needed here. You can start addressing my comments and we can meet early Jan and discuss this merge in person. what do you think?
| references: List[str] = [line.text for line in script_lines if line.speaker == "agent" and line.text is not None] | ||
| hypotheses: List[str] = [line.text for line in script_lines if line.speaker == "user" and line.text is not None] | ||
|
|
||
| if not references or not hypotheses: |
There was a problem hiding this comment.
Maybe you want to raise an Error here
There was a problem hiding this comment.
Updated evaluation.py
|
|
||
| Args: | ||
| script_lines (List[ScriptLine]): A list of script lines to evaluate. | ||
| metrics (List[str]): A list of metrics to use for evaluation. |
There was a problem hiding this comment.
Deep eval is legacy and should have been deleted but it must have copied over when I updated my branch
| if not references or not hypotheses: | ||
| return {"metrics": []} | ||
|
|
||
| metric_instance = Rouge() |
There was a problem hiding this comment.
is rouge the only metric implemented so far?
There was a problem hiding this comment.
No this was just a test implementation of evaluations.py
|
@batwood-1 thank you for addressing some of my comments. We will check this deeply tomorrow in person |
Description
Related Issue(s)
Motivation and Context
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist: