Merge pull request #886 from AntonOsika/refactor-poetry

The big refactor PR

.github/CONTRIBUTING.md

@@ -21,25 +21,22 @@ Please make sure to follow the established code style guidelines for this projec
 
 To enforce this we use [`pre-commit`](https://pre-commit.com/) to run [`black`](https://black.readthedocs.io/en/stable/index.html) and [`ruff`](https://beta.ruff.rs/docs/) on every commit.
 
-`pre-commit` is part of our `requirements.txt` file so you should already have it installed. If you don't, you can install the library via pip with:
+To install gpt-engineer as a developer, clone the repository and install the dependencies with:
 
 ```bash
-$ pip install -e .
-
-# For docs building, install doc dependencies too
-
-$ pip install -e .[doc]
+$ poetry install
+$ poetry shell
+```
 
-# And then install the `pre-commit` hooks with:
+And then install the `pre-commit` hooks with:
 
+```bash
 $ pre-commit install
 
 # output:
 pre-commit installed at .git/hooks/pre-commit
 ```
 
-Or you could just run `make dev-install` to install the dependencies and the hooks.
-
 If you are not familiar with the concept of [git hooks](https://git-scm.com/docs/githooks) and/or [`pre-commit`](https://pre-commit.com/) please read the documentation to understand how they work.
 
 As an introduction of the actual workflow, here is an example of the process you will encounter when you make a commit:

.github/workflows/ci.yaml

@@ -1,4 +1,4 @@
-name: Pip install and pytest
+name: Poetry install and pytest
 on:
   pull_request:
     branches: [ main ]
@@ -19,16 +19,19 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Install Poetry
+        run: |
+          pipx install poetry
+
       - uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-          cache: pip
-
-      - name: Install package
-        run: pip install -e .
+          cache: 'poetry'
 
-      - name: Install test runner
-        run: pip install pytest pytest-cov
+      # Install dependencies using Poetry
+      - name: Install dependencies
+        run: poetry install
 
+      # Run unit tests using Poetry
       - name: Run unit tests
-        run: pytest --cov=gpt_engineer
+        run: poetry run pytest --cov=gpt_engineer

.github/workflows/release.yaml

@@ -19,16 +19,33 @@ jobs:
       - uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
-          cache: pip
+          # Update to use Poetry's cache
+          cache: 'poetry'
 
-      - name: Install build tool
-        run: pip install build
+      # Install Poetry
+      - name: Install Poetry
+        run: |
+          curl -sSL https://install.python-poetry.org | python3 -
 
+      # Cache Poetry's dependencies based on the lock file
+      - name: Set up Poetry cache
+        uses: actions/cache@v3
+        with:
+          path: ~/.cache/pypoetry
+          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-poetry-
+
+      # Install dependencies using Poetry (if any)
+      - name: Install dependencies
+        run: poetry install
+
+      # Build package using Poetry
       - name: Build package
-        run: python -m build
+        run: poetry build --format sdist
 
-      - name: Upload package as build artifact
-        uses: actions/upload-artifact@v3
+      # Upload package as build artifact
+      - uses: actions/upload-artifact@v3
         with:
           name: package
           path: dist/
@@ -50,3 +67,5 @@ jobs:
 
       - name: Publish packages to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -51,13 +51,13 @@ scratchpad
 # Pyenv
 .python-version
 
-# Benchmark files
-benchmark
-!benchmark/*/prompt
-
 .gpte_consent
 
-.gpte_consent
+# projects folder apart from default prompt
+
+projects
+!projects/example/prompt
+!projects/example-improve
 
 # docs
 
@@ -72,8 +72,17 @@ docs/steps/
 docs/db
 docs/db/
 
-# poetry
-
-poetry.lock
+# coding assistants
 .aider*
 .gpteng
+
+# webapp specific
+webapp/node_modules
+webapp/package-lock.json
+
+webapp/.next/
+
+.langchain.db
+
+# TODO files
+/!todo*

Makefile

@@ -6,60 +6,46 @@ COLOR_RESET=\033[0m
 COLOR_CYAN=\033[1;36m
 COLOR_GREEN=\033[1;32m
 
-# Defines the targets help, install, dev-install, and run as phony targets. Phony targets are targets that are not really the name of files that are to be built. Instead, they are treated as commands.
+# Defines the targets help, install, dev-install, and run as phony targets.
 .PHONY: help install run
 
 #sets the default goal to help when no target is specified on the command line.
 .DEFAULT_GOAL := help
 
-#Disables echoing of commands. The commands executed by Makefile will not be printed on the console during execution.
+#Disables echoing of commands.
 .SILENT:
 
-#Sets the variable name to the second word from the MAKECMDGOALS. MAKECMDGOALS is a variable that contains the command-line targets specified when running make. In this case, the variable name will hold the value of the folder name specified when running the run target.
+#Sets the variable name to the second word from the MAKECMDGOALS.
 name := $(word 2,$(MAKECMDGOALS))
 
 #Defines a target named help.
 help:
 	@echo "Please use 'make <target>' where <target> is one of the following:"
 	@echo "  help           	Return this message with usage instructions."
-	@echo "  install        	Will install the dependencies and create a virtual environment."
+	@echo "  install        	Will install the dependencies using Poetry."
 	@echo "  run <folder_name>  Runs GPT Engineer on the folder with the given name."
 
-#Defines a target named install. This target will create a virtual environment, upgrade pip, install the dependencies, and install the pre-commit hooks. This means that running make install will first execute the create-venv target, then the upgrade-pip target, then the install-dependencies target, and finally the install-pre-commit target.
-install: create-venv upgrade-pip install-dependencies install-pre-commit farewell
+#Defines a target named install. This target will install the project using Poetry.
+install: poetry-install install-pre-commit farewell
 
-#Defines a target named create-venv. This target will create a virtual environment in the venv folder.
-create-venv:
-	@echo -e "$(COLOR_CYAN)Creating virtual environment...$(COLOR_RESET)" && \
-	python -m venv venv
-
-#Defines a target named upgrade-pip. This target will upgrade pip to the latest version.
-upgrade-pip:
-	@echo -e "$(COLOR_CYAN)Upgrading pip...$(COLOR_RESET)" && \
-	source venv/bin/activate && \
-	pip install --upgrade pip >> /dev/null
-
-#Defines a target named install-dependencies. This target will install the dependencies.
-install-dependencies:
-	@echo -e "$(COLOR_CYAN)Installing dependencies...$(COLOR_RESET)" && \
-	source venv/bin/activate && \
-	pip install -e . >> /dev/null
+#Defines a target named poetry-install. This target will install the project dependencies using Poetry.
+poetry-install:
+	@echo -e "$(COLOR_CYAN)Installing project with Poetry...$(COLOR_RESET)" && \
+	poetry install
 
 #Defines a target named install-pre-commit. This target will install the pre-commit hooks.
 install-pre-commit:
 	@echo -e "$(COLOR_CYAN)Installing pre-commit hooks...$(COLOR_RESET)" && \
-	source venv/bin/activate && \
-	pre-commit install
+	poetry run pre-commit install
 
 #Defines a target named farewell. This target will print a farewell message.
 farewell:
 	@echo -e "$(COLOR_GREEN)All done!$(COLOR_RESET)"
 
-#Defines a target named run. This target will run GPT Engineer on the folder with the given name, name was defined earlier in the Makefile.
+#Defines a target named run. This target will run GPT Engineer on the folder with the given name.
 run:
 	@echo -e "$(COLOR_CYAN)Running GPT Engineer on $(COLOR_GREEN)$(name)$(COLOR_CYAN) folder...$(COLOR_RESET)" && \
-	source venv/bin/activate && \
-	gpt-engineer projects/$(name)
+	poetry run gpt-engineer projects/$(name)
 
 # Counts the lines of code in the project
 cloc:

README.md

@@ -22,6 +22,8 @@ GPT Engineer is made to be easy to adapt, extend, and make your agent learn how
 
 ## Setup
 
+This project supports Python 3.8 - 3.11.
+
 Choose either **stable** or **development**.
 
 For **stable** release:
@@ -31,8 +33,11 @@ For **stable** release:
 For **development**:
 - `git clone https://github.com/AntonOsika/gpt-engineer.git`
 - `cd gpt-engineer`
-- `python -m pip install -e .`
-  - (or: `make install && source venv/bin/activate` for a venv)
+- `poetry install`
+- `poetry shell` to activate the virtual environment
+
+We have experimental support for [llama-index](https://github.com/run-llama/llama_index), [rank_bm25](https://github.com/dorianbrown/rank_bm25), and [tree_sitter_languages](https://github.com/grantjenks/py-tree-sitter-languages).
+- `poetry install --extras experimental`
 
 **API Key**
 
@@ -59,17 +64,21 @@ There are two ways to work with GPT-engineer: new code mode (the default), and i
 ### Creating new code
 - Create an empty folder for your project anywhere on your computer
 - Create a file called `prompt` (no extension) inside your new folder and fill it with instructions
-- Run `gpt-engineer <project_dir>` with a relative path to your folder
-  - For example: `gpt-engineer projects/my-new-project` from the gpt-engineer directory root with your new folder in `projects/`
+- Run `gpte <project_dir>` with a relative path to your folder
+  - For example: `gpte projects/my-new-project` from the gpt-engineer directory root with your new folder in `projects/`
 
 ### Improving Existing Code
 - Locate a folder with code which you want to improve anywhere on your computer
 - Create a file called `prompt` (no extension) inside your new folder and fill it with instructions for how you want to improve the code
-- Run `gpt-engineer <project_dir> -i` with a relative path to your folder
-  - For example: `gpt-engineer projects/my-old-project` from the gpt-engineer directory root with your folder in `projects/`
+- Run `gpte <project_dir> -i` with a relative path to your folder
+  - For example: `gpte projects/my-old-project` from the gpt-engineer directory root with your folder in `projects/`
 
 By running gpt-engineer you agree to our [terms](https://github.com/AntonOsika/gpt-engineer/blob/main/TERMS_OF_USE.md).
 
+### Note
+
+- To run this tool, the new command `gpte` is recommended for better user experience. However, the earlier default commands `gpt-engineer` and `ge` are also supported.
+
 
 ## Features
 

WINDOWS_README.md

@@ -29,8 +29,8 @@ For **development**:
 
 - `git clone [email protected]:AntonOsika/gpt-engineer.git`
 - `cd gpt-engineer`
-- `pip install -e .`
-  - (or: `make install && source venv/bin/activate` for a venv)
+- `poetry install`
+- `poetry shell` to activate the virtual environment
 
 ### Setup