Add dev container support for project (#704)

* Configure dev container for project

* Ignore personalization script

* enh(devcontainer): add pre-commit hooks

* enh(devcontainer): smaller image

* docs(devcontainer): Add instructions about how to use devcontainers

* enh(devcontainer): Configure pytest testing panel

* docs(devcontainer): Explain commit message regex enforcement

* enh(devcontainer): Add git lens extension

* enh(devcontainer): Fixes

* revert(devcontainer): Remove postgres CLI by default

* enh(devcontainer): Add tmux CLI tool

* fix(devcontainer): Exit if any commands in post create fail

* fix(devcontainer): Recompile lock file

* enh(devcontainer): Auto fetch from git

* fix(dependencies): Re-add pre-commit dep

* fix(devcontainers): Prevent failure when installing reqs with high num CPUs

* maint(python-version): Update github CI to 3.11

* ci(upgrade-poetry): Upgrade version in CI to reflect lock file version

* maint(devcontainer): Remove misspelled extension

* maint(devcontainer): Downgrade dev env for 3.9

* ci(devcontainer): Revert GH workflows back to 3.9

.devcontainer/devcontainer.json

@@ -0,0 +1,35 @@
+{
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "ms-vscode-remote.vscode-remote-extensionpack",
+        "charliermarsh.ruff",
+        "ms-azuretools.vscode-docker",
+        "ms-toolsai.jupyter",
+        "ms-python.mypy-type-checker",
+        "ms-vsliveshare.vsliveshare",
+        "ms-python.python",
+        "eamodio.gitlens",
+        "github.vscode-pull-request-github"
+      ],
+      "settings": {
+        "git.autofetch": true,
+        "python.testing.pytestEnabled": true,
+        "terminal.integrated.defaultProfile.linux": "zsh"
+      }
+    }
+  },
+  "features": {
+    "ghcr.io/devcontainers-contrib/features/tmux-apt-get:1": {},
+    "ghcr.io/devcontainers-contrib/features/zsh-plugins:0": {
+      "omzPlugins": "https://github.com/zsh-users/zsh-syntax-highlighting.git https://github.com/zsh-users/zsh-autosuggestions.git",
+      "plugins": "zsh-syntax-highlighting zsh-autosuggestions"
+    },
+    "ghcr.io/devcontainers/features/git:1": {},
+    "ghcr.io/devcontainers/features/github-cli:1": {},
+    "ghcr.io/schlich/devcontainer-features/powerlevel10k:1": {}
+  },
+  "image": "mcr.microsoft.com/devcontainers/python:3.9",
+  "name": "Python 3",
+  "postCreateCommand": "chmod +x ./.devcontainer/post_create.sh && ./.devcontainer/post_create.sh"
+}

.devcontainer/post_create.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+set -e  # Exit immediately if a command exits with a non-zero status.
+
+git config --global --add safe.directory /workspaces/dspy
+
+pip install poetry==1.7.1
+poetry config installer.max-workers 4
+
+poetry install --with dev
+
+sudo apt update
+sudo apt-get -y install python3-distutils
+
+poetry run pre-commit install --install-hooks
+
+personalization_script="./.devcontainer/.personalization.sh"
+
+# Developers can place a personalization script in the location specified above
+# to further customize their dev container
+if [ -f "$personalization_script" ]; then
+    echo "File $personalization_script exists. Running the script..."
+    chmod +x "$personalization_script"
+    $personalization_script
+fi

.github/workflows/run_tests.yml

@@ -8,7 +8,7 @@ on:
     types: [opened, synchronize, reopened]
 
 env:
-  POETRY_VERSION: "1.6.1"
+  POETRY_VERSION: "1.7.1"
 
 jobs:
   fix:

.gitignore

@@ -46,3 +46,4 @@ finetuning_ckpts/
 assertion.log
 *.log
 *.db
+/.devcontainer/.personalization.sh

CONTRIBUTING.md

@@ -1,19 +1,31 @@
 # Contributing
 
-## Finding Issues
+## Setting-up
 
-### Bounty Board
+### Preferred method - VSCode Dev Container
 
-The bounty board will have various features, issues, and requests that are up for grabs. We are still working on this. Come to the discord and ask for the current bounties.
+VSCode dev containers are a great way to containerize not only the necessary requirements but also recommended IDE extensions as well as settings such as pre-commit hooks and linting preferences. Using this will allow you to jump in to the perfect DSPY contribution environment without having to do much. Additionally, you'll be able to contribute through the web browser using Github Codespaces!
 
-See the spreadsheet [here](https://docs.google.com/spreadsheets/d/1psHSfFXENAxhQTd5veKRzKydVubD2Ov62aKQHiYC-CQ/edit?usp=sharing) for the current bounties.
+To use our dev container:
 
-## Setting-up
+1. Download Docker Desktop
+2. Download VSCode
+3. Within VSCode, install the Remote Development extension (ms-vscode-remote.vscode-remote-extensionpack)
+4. Open the VSCode command palette (cmd / ctrl + shift + p)
+5. Select `Dev Containers: Rebuild and Reopen in container`. A new VSCode window should open up and it should begin setting up your environment. Once it's done, you can open up a new terminal and start running tests or contributing!
+6. To test that your environment is set up correctly, open a new terminal and run the command `pytest`. You should be able to run all tests and see them pass. Alternatively, you can open up the testing panel, which looks like a beaker, and click the play button to run all of our tests.
+7. After the initial build, you should now be able to leave and re-enter the container any time without needing to rebuild. To do this, open the command palette and select `Dev Containers: Reopen in container`. This will not rebuild the container if you've done it correctly.
+
+NOTE: If you use this method, your default shell will be the poetry shell which will contain all the necessary requirements in your terminal. You shouldn't need to prefix python commands with poetry as you're already using the correct poetry virtual environment.
+
+### Alternative method
 
 To run the tests, you need to first clone the repository.
 
 Then install the package through poetry:
-Note - You may need to install poetry. See [here](https://python-poetry.org/docs/#installing-with-the-official-installer)
+Note - You may need to install poetry. You likely will just need to run `pip install poetry`. See [here](https://python-poetry.org/docs/#installing-with-the-official-installer)
+
+After installing poetry, use it to install our development requirements.
 
 ```bash
 poetry install --with dev
@@ -41,8 +53,26 @@ You may need the `--container-architecture linux/amd64` flag if you are on an M1
 
 Commit message format must be respected, with the following regex:
 
-This ends up looking like feature(dspy): added new feature
+This ends up looking like `feature(dspy): added new feature` or `enh(devcontainer): decreased size of image
 
 ```
 ^(break|build|ci|docs|feat|fix|perf|refactor|style|test|ops|hotfix|release|maint|init|enh|revert)\([a-z,A-Z,0-9,\-,\_,\/,:]+\)(:)\s{1}([\w\s]+)
 ```
+
+Detailed Breakdown
+^: Asserts the start of a line. This means the pattern must match from the beginning of the string.
+
+(break|build|ci|docs|feat|fix|perf|refactor|style|test|ops|hotfix|release|maint|init|enh|revert): This is a capture group that matches any one of the listed keywords. These keywords represent various types of commits, such as feat (feature), fix (bug fix), docs (documentation), etc.
+
+\( and \): Matches the literal parentheses ( and ). These are escaped with a backslash because parentheses are special characters in regular expressions, used for defining capture groups.
+
+[a-z,A-Z,0-9,\-,\_,\/,:]+: Matches one or more characters inside the square brackets. It includes lowercase and uppercase letters (a-z, A-Z), digits (0-9), and specific special characters (-, \_, /, :). The comma (,) here is likely intended as a separator in the explanation but is actually being treated as a literal character to match, which might be a mistake unless the comma is an expected character in this context.
+
+(:): Captures the colon character. This is another literal match, but it's also captured into a group because of the parentheses.
+
+\s{1}: Matches exactly one whitespace character. {1} is technically redundant since the default behavior without specifying a quantity is to match exactly one.
+
+([\w\s]+): This capture group matches one or more word characters (\w, which includes letters, digits, and underscores) or whitespace characters (\s). This part is likely intended to capture the commit message that follows the initial keyword and scope.
+
+Summary
+Putting it all together, this regex is used to enforce a structured format for commit messages, starting with a keyword indicating the commit type, followed by a scope enclosed in parentheses, a colon, a single space, and then the descriptive message. The scope part allows for various characters, including letters, numbers, and a few special characters, to accommodate different naming conventions.

pyproject.toml

@@ -121,6 +121,7 @@ torch = "^2.2.1"
 pytest-mock = "^3.12.0"
 ruff = "^0.3.0"
 black = "^24.2.0"
+pre-commit = "^3.7.0"
 
 [tool.poetry.extras]
 chromadb = ["chromadb"]