Add SQL Metrics Implementation #59
Conversation
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Changes requested. Reviewed everything up to da8a897 in 3 minutes and 30 seconds
More details
- Looked at
43lines of code in1files - Skipped
1files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_SbrHuyQL59T55GNV
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
| # Compare the formatted answer with each formatted ground truth answer | ||
| for formatted_gt in formatted_ground_truths: | ||
| # Simple string comparison for now, can be improved with more sophisticated methods | ||
| match_score = float(formatted_answer == formatted_gt) |
There was a problem hiding this comment.
Consider implementing a more sophisticated comparison method than simple string equality to handle cases where SQL queries might be functionally identical but differ in formatting or syntax. This could improve the robustness of the syntactic similarity evaluation.
There was a problem hiding this comment.
@ellipsis-dev come up with a few more sophisticated ways to handle functionally identical cases
|
This is a cross repository pull request, but Ellipsis isn't installed in yisz/continuous-eval-locale. In order to have Ellipsis address comments in this PR, you'll need to install Ellipsis in that repository. |
Add sql metrics
|
@pantonante check to see if Devin's work is good enough. It added tests / documentation as well. |
| It uses the sqlparse library to format and compare the SQL queries. | ||
| """ | ||
|
|
||
| def __call__(self, answer: str, ground_truth_answers: Union[List[str], str]): |
There was a problem hiding this comment.
This would be better I think: https://github.com/tobymao/sqlglot?tab=readme-ov-file#ast-diff
Pull Request Description
Summary
This pull request introduces a new SQL AST comparison metric to the
continuous-evalrepository. The new metric,SQLASTSimilarity, compares SQL queries using Abstract Syntax Tree (AST) similarity, leveraging thesqlglotlibrary.Changes
SQLASTSimilarityclass to thecode_deterministic_metrics.pyfile.diffandparse_onefunctions from thesqlglotlibrary.Keepclass from thesqlglot.diffmodule.__call__method in theSQLASTSimilarityclass to parse SQL queries into ASTs and calculate similarity scores._calculate_similaritymethod in theSQLASTSimilarityclass to calculate the similarity score between two ASTs by using thedifffunction to get the differences between the trees, counting the total changes, and calculating the total number of nodes in both trees. The similarity score is calculated as1 - (total_changes / total_nodes).Testing
test_code_deterministic_metrics.py, with unit tests for theSQLASTSimilarityclass.SQLASTSimilarityclass, including tests for exact match, different queries, similar queries, and invalid queries.pytest, and all tests passed successfully.Link to Devin run
https://preview.devin.ai/devin/696032ba45654233968d6a04f2bc5df3
Request for Review
Please review the changes and provide feedback. If everything looks good, kindly approve the pull request for merging.
Thank you!