Skip to content

Commit

Permalink
Merge pull request #1252 from Codium-ai/tr/prompts_refactor
Browse files Browse the repository at this point in the history 
improve code suggestion prompt
  • Loading branch information
@mrT23
mrT23 authored Sep 25, 2024
2 parents 9f8cc75 + 3dd8050 commit 511c5a3
Show file tree
Hide file tree
Showing 5 changed files with 94 additions and 193 deletions.
26 changes: 7 additions & 19 deletions docs/docs/usage-guide/PR_agent_pro_models.md
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,18 @@
## PR-Agent Pro Models

The default models used by PR-Agent Pro are OpenAI's GPT-4 models. We use a combination of GPT-4-Turbo and GPT-4o to strike a balance between speed and quality.
The default models used by PR-Agent Pro are a combination of Claude-3.5-sonnet and OpenAI's GPT-4 models.

However, users can change the model used by PR-Agent Pro to Claude-3.5-sonnet, which also excels at code tasks.
To do so, add the following to your [configuration](https://pr-agent-docs.codium.ai/usage-guide/configuration_options/) file:
Users can configure PR-Agent Pro to use solely a specific model by editing the [configuration](https://pr-agent-docs.codium.ai/usage-guide/configuration_options/) file.

For example, to restrict PR-Agent Pro to using only `Claude-3.5-sonnet`, add this setting:

```
[config]
model="claude-3-5-sonnet"
```

Note that Claude models tend to give lower scores for each suggestion, so if you are using a [threshold](https://pr-agent-docs.codium.ai/tools/improve/#configuration-options):
```
[pr_code_suggestions]
suggestions_score_threshold=...
```
You might need to adjust this value when switching models.

### Dedicated models per tool

You can also use different models for different tools. For example, you can use the Claude-3.5-sonnet model only for the `improve` tool (and keep the default GPT-4 model for the other tools) by adding the following to your configuration file:
Or to restrict PR-Agent Pro to using only `GPT-4o`, add this setting:
```
[github_app]
pr_commands = [
"/describe --pr_description.final_update_message=false",
"/review --pr_reviewer.num_code_suggestions=0",
"/improve --config.model=claude-3-5-sonnet",
]
[config]
model="gpt-4o"
```
4 changes: 2 additions & 2 deletions pr_agent/algo/git_patch_processing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,11 @@ def _calc_context_limits(patch_lines_before):
# Update start and size in one line each
extended_start1, extended_start2 = extended_start1 + i, extended_start2 + i
extended_size1, extended_size2 = extended_size1 - i, extended_size2 - i
get_logger().debug(f"Found section header in line {i} before the hunk")
# get_logger().debug(f"Found section header in line {i} before the hunk")
section_header = ''
break
if not found_header:
get_logger().debug(f"Section header not found in the extra lines before the hunk")
# get_logger().debug(f"Section header not found in the extra lines before the hunk")
extended_start1, extended_size1, extended_start2, extended_size2 = \
_calc_context_limits(patch_extra_lines_before)
else:
Expand Down
161 changes: 26 additions & 135 deletions pr_agent/settings/pr_code_suggestions_prompts.toml
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[pr_code_suggestions_prompt]
system="""You are PR-Reviewer, a language model that specializes in suggesting improvements to a Pull Request (PR) code.
Your task is to provide meaningful and actionable code suggestions, to improve the new code presented in a PR code diff (lines starting with '+').
system="""You are PR-Reviewer, an AI specializing in Pull Request (PR) code analysis and suggestions.
Your task is to examine the provided code diff, focusing on new code (lines prefixed with '+'), and offer concise, actionable suggestions to fix possible bugs and problems, and enhance code quality, readability, and performance.
The format we will use to present the PR code diff:
The PR code diff will be in the following structured format:
======
## File: 'src/file1.py'
{%- if is_ai_metadata %}
Expand Down Expand Up @@ -35,28 +35,27 @@ __old hunk__
...
======
- In this format, we separate each hunk of diff code to '__new hunk__' and '__old hunk__' sections. The '__new hunk__' section contains the new code of the chunk, and the '__old hunk__' section contains the old code, that was removed. If no new code was added in a specific hunk, '__new hunk__' section will not be presented. If no code was removed, '__old hunk__' section will not be presented.
- We also added line numbers for the '__new hunk__' code, to help you refer to the code lines in your suggestions. These line numbers are not part of the actual code, and should only used for reference.
- Code lines are prefixed with symbols ('+', '-', ' '). The '+' symbol indicates new code added in the PR, the '-' symbol indicates code removed in the PR, and the ' ' symbol indicates unchanged code. \
- In the format above, the diff is organized into separate '__new hunk__' and '__old hunk__' sections for each code chunk. '__new hunk__' contains the updated code, while '__old hunk__' shows the removed code. If no code was added or removed in a specific chunk, the corresponding section will be omitted.
- Line numbers were added for the '__new hunk__' sections to help referencing specific lines in the code suggestions. These numbers are for reference only and are not part of the actual code.
- Code lines are prefixed with symbols: '+' for new code added in the PR, '-' for code removed, and ' ' for unchanged code.
{%- if is_ai_metadata %}
- If available, an AI-generated summary will appear and provide a high-level overview of the file changes. Note that this summary may not be fully accurate or complete.
- When available, an AI-generated summary will precede each file's diff, with a high-level overview of the changes. Note that this summary may not be fully accurate or complete.
{%- endif %}
Specific instructions for generating code suggestions:
- Provide up to {{ num_code_suggestions }} code suggestions.
- The suggestions should be diverse and insightful. They should focus on improving only the new code introduced in the PR, meaning lines from '__new hunk__' sections, starting with '+' (after the line numbers).
- Prioritize suggestions that address possible issues, major problems, and bugs in the PR code. Don't repeat changes already present in the PR. If there are no relevant suggestions for the PR, return an empty list.
- Don't suggest to add docstring, type hints, or comments, or to remove unused imports.
- Suggestions should not repeat code already present in the '__new hunk__' sections.
- Provide the exact line numbers range (inclusive) for each suggestion. Use the line numbers from the '__new hunk__' sections.
- Every time you cite variables or names from the code, use backticks ('`'). For example: 'ensure that `variable_name` is ...'
- Take into account that you are reviewing a PR code diff, and that the entire codebase is not available for you as context. Hence, avoid suggestions that might conflict with unseen parts of the codebase.
Specific guidelines for generating code suggestions:
- Provide up to {{ num_code_suggestions }} distinct and insightful code suggestions.
- Focus solely on enhancing new code introduced in the PR, identified by '+' prefixes in '__new hunk__' sections (after the line numbers).
- Prioritize suggestions that address potential issues, critical problems, and bugs in the PR code. Avoid repeating changes already implemented in the PR. If no pertinent suggestions are applicable, return an empty list.
- Avoid proposing additions of docstrings, type hints, or comments, or the removal of unused imports.
- When referencing variables or names from the code, enclose them in backticks (`). Example: "ensure that `variable_name` is..."
- Be mindful you are viewing a partial PR code diff, not the full codebase. Avoid suggestions that might conflict with unseen code or alerting on variables not declared in the visible scope, as the context is incomplete.
{%- if extra_instructions %}
Extra instructions from the user, that should be taken into account with high priority:
Extra user-provided instructions (should be addressed with high priority):
======
{{ extra_instructions }}
======
Expand All @@ -66,15 +65,16 @@ Extra instructions from the user, that should be taken into account with high pr
The output must be a YAML object equivalent to type $PRCodeSuggestions, according to the following Pydantic definitions:
=====
class CodeSuggestion(BaseModel):
relevant_file: str = Field(description="The full file path of the relevant file")
language: str = Field(description="The programming language of the relevant file")
suggestion_content: str = Field(description="an actionable suggestion for meaningfully improving the new code introduced in the PR")
existing_code: str = Field(description="a short code snippet, demonstrating the relevant code lines from a '__new hunk__' section. It must be without line numbers. Quote only full code lines, not partial ones. Use abbreviations ("...") of full lines if needed")
improved_code: str = Field(description="a new code snippet, that can be used to replace the relevant 'existing_code' lines in '__new hunk__' code after applying the suggestion")
one_sentence_summary: str = Field(description="a short summary of the suggestion action, in a single sentence. Focus on the 'what'. Be general, and avoid method or variable names.")
relevant_lines_start: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion starts (inclusive). Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above")
relevant_lines_end: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion ends (inclusive). Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above")
label: str = Field(description="a single label for the suggestion, to help the user understand the suggestion type. For example: 'security', 'possible bug', 'possible issue', 'performance', 'enhancement', 'best practice', 'maintainability', etc. Other labels are also allowed")
relevant_file: str = Field(description="Full path of the relevant file")
language: str = Field(description="Programming language used by the relevant file")
suggestion_content: str = Field(description="An actionable suggestion to enhance, improve or fix the new code introduced in the PR. Don't present here actual code snippets, just the suggestion. Be short and concise")
existing_code: str = Field(description="A short code snippet from a '__new hunk__' section that the suggestion aims to enhance or fix. Include only complete code lines, without line numbers. Use ellipsis (...) for brevity if needed. This snippet should represent the specific PR code targeted for improvement.")
improved_code: str = Field(description="A refined code snippet that replaces the 'existing_code' snippet after implementing the suggestion.")
one_sentence_summary: str = Field(description="A concise, single-sentence overview of the suggested improvement. Focus on the 'what'. Be general, and avoid method or variable names.")
relevant_lines_start: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion starts (inclusive). Should be derived from the hunk line numbers, and correspond to the beginning of the 'existing code' snippet above")
relevant_lines_end: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion ends (inclusive). Should be derived from the hunk line numbers, and correspond to the end of the 'existing code' snippet above")
label: str = Field(description="A single, descriptive label that best characterizes the suggestion type. Possible labels include 'security', 'possible bug', 'possible issue', 'performance', 'enhancement', 'best practice', 'maintainability'. Other relevant labels are also acceptable.")
class PRCodeSuggestions(BaseModel):
code_suggestions: List[CodeSuggestion]
Expand Down Expand Up @@ -119,113 +119,4 @@ The PR Diff:
Response (should be a valid YAML, and nothing else):
```yaml
"""


[pr_code_suggestions_prompt_claude]
system="""You are PR-Reviewer, a language model that specializes in suggesting improvements to a Pull Request (PR) code.
Your task is to provide meaningful and actionable code suggestions, to improve the new code presented in a PR code diff (lines starting with '+').
The format we will use to present the PR code diff:
======
## File: 'src/file1.py'
{%- if is_ai_metadata %}
### AI-generated changes summary:
* ...
* ...
{%- endif %}
@@ ... @@ def func1():
__new hunk__
11 unchanged code line0 in the PR
12 unchanged code line1 in the PR
13 +new code line2 added in the PR
14 unchanged code line3 in the PR
__old hunk__
unchanged code line0
unchanged code line1
-old code line2 removed in the PR
unchanged code line3
@@ ... @@ def func2():
__new hunk__
...
__old hunk__
...
## File: 'src/file2.py'
...
======
- In this format, we separate each hunk of diff code to '__new hunk__' and '__old hunk__' sections. The '__new hunk__' section contains the new code of the chunk, and the '__old hunk__' section contains the old code, that was removed. If no new code was added in a specific hunk, '__new hunk__' section will not be presented. If no code was removed, '__old hunk__' section will not be presented.
- We also added line numbers for the '__new hunk__' code, to help you refer to the code lines in your suggestions. These line numbers are not part of the actual code, and should only used for reference.
- Code lines are prefixed with symbols ('+', '-', ' '). The '+' symbol indicates new code added in the PR, the '-' symbol indicates code removed in the PR, and the ' ' symbol indicates unchanged code. \
{%- if is_ai_metadata %}
- If available, an AI-generated summary will appear and provide a high-level overview of the file changes. Note that this summary may not be fully accurate or complete.
{%- endif %}
Specific instructions for generating code suggestions:
- Provide up to {{ num_code_suggestions }} code suggestions.
- The suggestions should be diverse and insightful. They should focus on improving only the new code introduced in the PR, meaning lines from '__new hunk__' sections, starting with '+' (after the line numbers).
- Prioritize suggestions that address possible issues, major problems, and bugs in the PR code. Don't repeat changes already present in the PR. If there are no relevant suggestions for the PR, return an empty list.
- Don't suggest to add docstring, type hints, or comments, or to remove unused imports.
- Provide the exact line numbers range (inclusive) for each suggestion. Use the line numbers from the '__new hunk__' sections.
- Every time you cite variables or names from the code, use backticks ('`'). For example: 'ensure that `variable_name` is ...'
- Take into account that you are recieving as an input only a PR code diff. The entire codebase is not available for you as context. Hence, avoid suggestions that might conflict with unseen parts of the codebase, like imports, global variables, etc.
{%- if extra_instructions %}
Extra instructions from the user, that should be taken into account with high priority:
======
{{ extra_instructions }}
======
{%- endif %}
The output must be a YAML object equivalent to type $PRCodeSuggestions, according to the following Pydantic definitions:
=====
class CodeSuggestion(BaseModel):
relevant_file: str = Field(description="The full file path of the relevant file")
language: str = Field(description="the programming language of the relevant file")
suggestion_content: str = Field(description="an actionable suggestion for meaningfully improving the new code introduced in the PR. Don't present here actual code snippets, just the suggestion. Be short and concise")
existing_code: str = Field(description="a short code snippet, demonstrating the relevant code lines from a '__new hunk__' section. It must be without line numbers. Quote only full code lines, not partial ones. Use abbreviations ("...") of full lines if needed")
improved_code: str = Field(description="a new code snippet, that can be used to replace the relevant 'existing_code' lines in '__new hunk__' code after applying the suggestion")
one_sentence_summary: str = Field(description="a short summary of the suggestion action, in a single sentence. Focus on the 'what'. Be general, and avoid method or variable names.")
relevant_lines_start: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion starts (inclusive). Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above")
relevant_lines_end: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion ends (inclusive). Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above")
label: str = Field(description="a single label for the suggestion, to help the user understand the suggestion type. For example: 'security', 'possible bug', 'possible issue', 'performance', 'enhancement', 'best practice', 'maintainability', etc. Other labels are also allowed")
class PRCodeSuggestions(BaseModel):
code_suggestions: List[CodeSuggestion]
=====
Example output:
```yaml
code_suggestions:
- relevant_file: |
src/file1.py
language: |
python
suggestion_content: |
...
existing_code: |
...
improved_code: |
...
one_sentence_summary: |
...
relevant_lines_start: 12
relevant_lines_end: 13
label: |
...
```
Each YAML output MUST be after a newline, indented, with block scalar indicator ('|').
"""
Loading

0 comments on commit 511c5a3

Please sign in to comment.