From a03c7f5c0daac6b6dac6e46c2ee1b045dcf81c7c Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 11:32:44 -0800 Subject: [PATCH 01/13] contributor docs --- .github/CONTRIBUTING.md | 376 +----------------- docs/.local_build.sh | 2 +- docs/docs/contributing/code.mdx | 261 ++++++++++++ docs/docs/contributing/documentation.mdx | 46 +++ docs/docs/contributing/index.mdx | 44 ++ docs/docs/contributing/integrations.mdx | 6 + .../contributing/reference/_category_.yml | 3 + docs/docs/contributing/reference/packages.mdx | 49 +++ docs/sidebars.js | 11 + docs/vercel_build.sh | 1 - 10 files changed, 422 insertions(+), 377 deletions(-) create mode 100644 docs/docs/contributing/code.mdx create mode 100644 docs/docs/contributing/documentation.mdx create mode 100644 docs/docs/contributing/index.mdx create mode 100644 docs/docs/contributing/integrations.mdx create mode 100644 docs/docs/contributing/reference/_category_.yml create mode 100644 docs/docs/contributing/reference/packages.mdx diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 3a8a7ebd9be3f..bab7857998d85 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -3,378 +3,4 @@ Hi there! Thank you for even being interested in contributing to LangChain. As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes. -## 🗺️ Guidelines - -### 👩‍💻 Contributing Code - -To contribute to this project, please follow the ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow. -Please do not try to push directly to this repo unless you are a maintainer. - -Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant -maintainers. - -Pull requests cannot land without passing the formatting, linting, and testing checks first. See [Testing](#testing) and -[Formatting and Linting](#formatting-and-linting) for how to run these checks locally. - -It's essential that we maintain great documentation and testing. If you: -- Fix a bug - - Add a relevant unit or integration test when possible. These live in `tests/unit_tests` and `tests/integration_tests`. -- Make an improvement - - Update any affected example notebooks and documentation. These live in `docs`. - - Update unit and integration tests when relevant. -- Add a feature - - Add a demo notebook in `docs/docs/`. - - Add unit and integration tests. - -We are a small, progress-oriented team. If there's something you'd like to add or change, opening a pull request is the -best way to get our attention. - -### 🚩GitHub Issues - -Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests. - -There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues. - -If you start working on an issue, please assign it to yourself. - -If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature. -If two issues are related, or blocking, please link them rather than combining them. - -We will try to keep these issues as up-to-date as possible, though -with the rapid rate of development in this field some may get out of date. -If you notice this happening, please let us know. - -### 🙋Getting Help - -Our goal is to have the simplest developer setup possible. Should you experience any difficulty getting setup, please -contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is -smooth for future contributors. - -In a similar vein, we do enforce certain linting, formatting, and documentation standards in the codebase. -If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help - -we do not want these to get in the way of getting good code into the codebase. - -## 🚀 Quick Start - -This quick start guide explains how to run the repository locally. -For a [development container](https://containers.dev/), see the [.devcontainer folder](https://github.com/langchain-ai/langchain/tree/master/.devcontainer). - -### Dependency Management: Poetry and other env/dependency managers - -This project utilizes [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager. - -❗Note: *Before installing Poetry*, if you use `Conda`, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`) - -Install Poetry: **[documentation on how to install it](https://python-poetry.org/docs/#installation)**. - -❗Note: If you use `Conda` or `Pyenv` as your environment/package manager, after installing Poetry, -tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`) - -### Different packages - -This repository contains multiple packages: -- `langchain-core`: Base interfaces for key abstractions as well as logic for combining them in chains (LangChain Expression Language). -- `langchain-community`: Third-party integrations of various components. -- `langchain`: Chains, agents, and retrieval logic that makes up the cognitive architecture of your applications. -- `langchain-experimental`: Components and chains that are experimental, either in the sense that the techniques are novel and still being tested, or they require giving the LLM more access than would be possible in most production systems. - -Each of these has its own development environment. Docs are run from the top-level makefile, but development -is split across separate test & release flows. - -For this quickstart, start with langchain: - -```bash -cd libs/langchain -``` - -### Local Development Dependencies - -Install langchain development requirements (for running langchain, running examples, linting, formatting, tests, and coverage): - -```bash -poetry install --with test -``` - -Then verify dependency installation: - -```bash -make test -``` - -If the tests don't pass, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`. - -If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running -Poetry v1.6.1+. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases. -If you are still seeing this bug on v1.6.1, you may also try disabling "modern installation" -(`poetry config installer.modern-installation false`) and re-installing requirements. -See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details. - -### Testing - -_some test dependencies are optional; see section about optional dependencies_. - -Unit tests cover modular logic that does not require calls to outside APIs. -If you add new logic, please add a unit test. - -To run unit tests: - -```bash -make test -``` - -To run unit tests in Docker: - -```bash -make docker_tests -``` - -There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available. - -### Only develop langchain_core or langchain_experimental - -If you are only developing `langchain_core` or `langchain_experimental`, you can simply install the dependencies for the respective projects and run tests: - -```bash -cd libs/core -poetry install --with test -make test -``` - -Or: - -```bash -cd libs/experimental -poetry install --with test -make test -``` - -### Formatting and Linting - -Run these locally before submitting a PR; the CI system will check also. - -#### Code Formatting - -Formatting for this project is done via [ruff](https://docs.astral.sh/ruff/rules/). - -To run formatting for docs, cookbook and templates: - -```bash -make format -``` - -To run formatting for a library, run the same command from the relevant library directory: - -```bash -cd libs/{LIBRARY} -make format -``` - -Additionally, you can run the formatter only on the files that have been modified in your current branch as compared to the master branch using the format_diff command: - -```bash -make format_diff -``` - -This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase. - -#### Linting - -Linting for this project is done via a combination of [ruff](https://docs.astral.sh/ruff/rules/) and [mypy](http://mypy-lang.org/). - -To run linting for docs, cookbook and templates: - -```bash -make lint -``` - -To run linting for a library, run the same command from the relevant library directory: - -```bash -cd libs/{LIBRARY} -make lint -``` - -In addition, you can run the linter only on the files that have been modified in your current branch as compared to the master branch using the lint_diff command: - -```bash -make lint_diff -``` - -This can be very helpful when you've made changes to only certain parts of the project and want to ensure your changes meet the linting standards without having to check the entire codebase. - -We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed. - -#### Spellcheck - -Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell). -Note that `codespell` finds common typos, so it could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words. - -To check spelling for this project: - -```bash -make spell_check -``` - -To fix spelling in place: - -```bash -make spell_fix -``` - -If codespell is incorrectly flagging a word, you can skip spellcheck for that word by adding it to the codespell config in the `pyproject.toml` file. - -```python -[tool.codespell] -... -# Add here: -ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure' -``` - -## Working with Optional Dependencies - -Langchain relies heavily on optional dependencies to keep the Langchain package lightweight. - -You only need to add a new dependency if a **unit test** relies on the package. -If your package is only required for **integration tests**, then you can skip these -steps and leave all pyproject.toml and poetry.lock files alone. - -If you're adding a new dependency to Langchain, assume that it will be an optional dependency, and -that most users won't have it installed. - -Users who do not have the dependency installed should be able to **import** your code without -any side effects (no warnings, no errors, no exceptions). - -To introduce the dependency to the pyproject.toml file correctly, please do the following: - -1. Add the dependency to the main group as an optional dependency - ```bash - poetry add --optional [package_name] - ``` -2. Open pyproject.toml and add the dependency to the `extended_testing` extra -3. Relock the poetry file to update the extra. - ```bash - poetry lock --no-update - ``` -4. Add a unit test that the very least attempts to import the new code. Ideally, the unit -test makes use of lightweight fixtures to test the logic of the code. -5. Please use the `@pytest.mark.requires(package_name)` decorator for any tests that require the dependency. - -## Adding a Jupyter Notebook - -If you are adding a Jupyter Notebook example, you'll want to install the optional `dev` dependencies. - -To install dev dependencies: - -```bash -poetry install --with dev -``` - -Launch a notebook: - -```bash -poetry run jupyter notebook -``` - -When you run `poetry install`, the `langchain` package is installed as editable in the virtualenv, so your new logic can be imported into the notebook. - -## Documentation - -While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing. -This covers how to get started contributing to documentation. - -From the top-level of this repo, install documentation dependencies: - -```bash -poetry install -``` - -### Contribute Documentation - -The docs directory contains Documentation and API Reference. - -Documentation is built using [Docusaurus 2](https://docusaurus.io/). - -API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code. -For that reason, we ask that you add good documentation to all classes and methods. - -Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed. - -### Build Documentation Locally - -In the following commands, the prefix `api_` indicates that those are operations for the API Reference. - -Before building the documentation, it is always a good idea to clean the build directory: - -```bash -make docs_clean -make api_docs_clean -``` - -Next, you can build the documentation as outlined below: - -```bash -make docs_build -make api_docs_build -``` - -Finally, run the link checker to ensure all links are valid: - -```bash -make docs_linkcheck -make api_docs_linkcheck -``` - -### Verify Documentation changes - -After pushing documentation changes to the repository, you can preview and verify that the changes are -what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page. -This will take you to a preview of the documentation changes. -This preview is created by [Vercel](https://vercel.com/docs/getting-started-with-vercel). - -## 📕 Releases & Versioning - -As of now, LangChain has an ad hoc release process: releases are cut with high frequency by -a maintainer and published to [PyPI](https://pypi.org/). -The different packages are versioned slightly differently. - -### `langchain-core` - -`langchain-core` is currently on version `0.1.x`. - -As `langchain-core` contains the base abstractions and runtime for the whole LangChain ecosystem, we will communicate any breaking changes with advance notice and version bumps. The exception for this is anything in `langchain_core.beta`. The reason for `langchain_core.beta` is that given the rate of change of the field, being able to move quickly is still a priority, and this module is our attempt to do so. - -Minor version increases will occur for: - -- Breaking changes for any public interfaces NOT in `langchain_core.beta` - -Patch version increases will occur for: - -- Bug fixes -- New features -- Any changes to private interfaces -- Any changes to `langchain_core.beta` - -### `langchain` - -`langchain` is currently on version `0.0.x` - -All changes will be accompanied by a patch version increase. Any changes to public interfaces are nearly always done in a backwards compatible way and will be communicated ahead of time when they are not backwards compatible. - -We are targeting January 2024 for a release of `langchain` v0.1, at which point `langchain` will adopt the same versioning policy as `langchain-core`. - -### `langchain-community` - -`langchain-community` is currently on version `0.0.x` - -All changes will be accompanied by a patch version increase. - -### `langchain-experimental` - -`langchain-experimental` is currently on version `0.0.x` - -All changes will be accompanied by a patch version increase. - -## 🌟 Recognition - -If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)! -If you have a Twitter account you would like us to mention, please let us know in the PR or through another means. +To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/) \ No newline at end of file diff --git a/docs/.local_build.sh b/docs/.local_build.sh index 9a3f9a79f82d6..77b604ededcb7 100755 --- a/docs/.local_build.sh +++ b/docs/.local_build.sh @@ -13,9 +13,9 @@ rsync -ruv --exclude node_modules --exclude api_reference --exclude .venv --excl cd ../_dist poetry run python scripts/model_feat_table.py cp ../cookbook/README.md src/pages/cookbook.mdx -cp ../.github/CONTRIBUTING.md docs/contributing.md mkdir -p docs/templates cp ../templates/docs/INDEX.md docs/templates/index.md +poetry run python scripts/copy_templates.py wget https://raw.githubusercontent.com/langchain-ai/langserve/main/README.md -O docs/langserve.md yarn diff --git a/docs/docs/contributing/code.mdx b/docs/docs/contributing/code.mdx new file mode 100644 index 0000000000000..c20a0772b2a59 --- /dev/null +++ b/docs/docs/contributing/code.mdx @@ -0,0 +1,261 @@ +--- +sidebar_label: Code +sidebar_position: 2 +--- +# Contribute Code + +To contribute to this project, please follow the ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow. +Please do not try to push directly to this repo unless you are a maintainer. + +Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant +maintainers. + +Pull requests cannot land without passing the formatting, linting, and testing checks first. See [Testing](#testing) and +[Formatting and Linting](#formatting-and-linting) for how to run these checks locally. + +It's essential that we maintain great documentation and testing. If you: +- Fix a bug + - Add a relevant unit or integration test when possible. These live in `tests/unit_tests` and `tests/integration_tests`. +- Make an improvement + - Update any affected example notebooks and documentation. These live in `docs`. + - Update unit and integration tests when relevant. +- Add a feature + - Add a demo notebook in `docs/docs/`. + - Add unit and integration tests. + +We are a small, progress-oriented team. If there's something you'd like to add or change, opening a pull request is the +best way to get our attention. + +## 🚀 Quick Start + +This quick start guide explains how to run the repository locally. +For a [development container](https://containers.dev/), see the [.devcontainer folder](https://github.com/langchain-ai/langchain/tree/master/.devcontainer). + +### Dependency Management: Poetry and other env/dependency managers + +This project utilizes [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager. + +❗Note: *Before installing Poetry*, if you use `Conda`, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`) + +Install Poetry: **[documentation on how to install it](https://python-poetry.org/docs/#installation)**. + +❗Note: If you use `Conda` or `Pyenv` as your environment/package manager, after installing Poetry, +tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`) + +### Different packages + +This repository contains multiple packages: +- `langchain-core`: Base interfaces for key abstractions as well as logic for combining them in chains (LangChain Expression Language). +- `langchain-community`: Third-party integrations of various components. +- `langchain`: Chains, agents, and retrieval logic that makes up the cognitive architecture of your applications. +- `langchain-experimental`: Components and chains that are experimental, either in the sense that the techniques are novel and still being tested, or they require giving the LLM more access than would be possible in most production systems. + +Each of these has its own development environment. Docs are run from the top-level makefile, but development +is split across separate test & release flows. + +For this quickstart, start with langchain: + +```bash +cd libs/langchain +``` + +### Local Development Dependencies + +Install langchain development requirements (for running langchain, running examples, linting, formatting, tests, and coverage): + +```bash +poetry install --with test +``` + +Then verify dependency installation: + +```bash +make test +``` + +If the tests don't pass, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`. + +If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running +Poetry v1.6.1+. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases. +If you are still seeing this bug on v1.6.1, you may also try disabling "modern installation" +(`poetry config installer.modern-installation false`) and re-installing requirements. +See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details. + +### Testing + +_some test dependencies are optional; see section about optional dependencies_. + +Unit tests cover modular logic that does not require calls to outside APIs. +If you add new logic, please add a unit test. + +To run unit tests: + +```bash +make test +``` + +To run unit tests in Docker: + +```bash +make docker_tests +``` + +There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available. + +### Only develop langchain_core or langchain_experimental + +If you are only developing `langchain_core` or `langchain_experimental`, you can simply install the dependencies for the respective projects and run tests: + +```bash +cd libs/core +poetry install --with test +make test +``` + +Or: + +```bash +cd libs/experimental +poetry install --with test +make test +``` + +### Formatting and Linting + +Run these locally before submitting a PR; the CI system will check also. + +#### Code Formatting + +Formatting for this project is done via [ruff](https://docs.astral.sh/ruff/rules/). + +To run formatting for docs, cookbook and templates: + +```bash +make format +``` + +To run formatting for a library, run the same command from the relevant library directory: + +```bash +cd libs/{LIBRARY} +make format +``` + +Additionally, you can run the formatter only on the files that have been modified in your current branch as compared to the master branch using the format_diff command: + +```bash +make format_diff +``` + +This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase. + +#### Linting + +Linting for this project is done via a combination of [ruff](https://docs.astral.sh/ruff/rules/) and [mypy](http://mypy-lang.org/). + +To run linting for docs, cookbook and templates: + +```bash +make lint +``` + +To run linting for a library, run the same command from the relevant library directory: + +```bash +cd libs/{LIBRARY} +make lint +``` + +In addition, you can run the linter only on the files that have been modified in your current branch as compared to the master branch using the lint_diff command: + +```bash +make lint_diff +``` + +This can be very helpful when you've made changes to only certain parts of the project and want to ensure your changes meet the linting standards without having to check the entire codebase. + +We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed. + +#### Spellcheck + +Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell). +Note that `codespell` finds common typos, so it could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words. + +To check spelling for this project: + +```bash +make spell_check +``` + +To fix spelling in place: + +```bash +make spell_fix +``` + +If codespell is incorrectly flagging a word, you can skip spellcheck for that word by adding it to the codespell config in the `pyproject.toml` file. + +```python +[tool.codespell] +... +# Add here: +ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure' +``` + +## Working with Optional Dependencies + +Langchain relies heavily on optional dependencies to keep the Langchain package lightweight. + +You only need to add a new dependency if a **unit test** relies on the package. +If your package is only required for **integration tests**, then you can skip these +steps and leave all pyproject.toml and poetry.lock files alone. + +If you're adding a new dependency to Langchain, assume that it will be an optional dependency, and +that most users won't have it installed. + +Users who do not have the dependency installed should be able to **import** your code without +any side effects (no warnings, no errors, no exceptions). + +To introduce the dependency to the pyproject.toml file correctly, please do the following: + +1. Add the dependency to the main group as an optional dependency + ```bash + poetry add --optional [package_name] + ``` +2. Open pyproject.toml and add the dependency to the `extended_testing` extra +3. Relock the poetry file to update the extra. + ```bash + poetry lock --no-update + ``` +4. Add a unit test that the very least attempts to import the new code. Ideally, the unit +test makes use of lightweight fixtures to test the logic of the code. +5. Please use the `@pytest.mark.requires(package_name)` decorator for any tests that require the dependency. + +## Adding a Jupyter Notebook + +If you are adding a Jupyter Notebook example, you'll want to install the optional `dev` dependencies. + +To install dev dependencies: + +```bash +poetry install --with dev +``` + +Launch a notebook: + +```bash +poetry run jupyter notebook +``` + +When you run `poetry install`, the `langchain` package is installed as editable in the virtualenv, so your new logic can be imported into the notebook. + +## Documentation + +While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing. +This covers how to get started contributing to documentation. + +From the top-level of this repo, install documentation dependencies: + +```bash +poetry install +``` \ No newline at end of file diff --git a/docs/docs/contributing/documentation.mdx b/docs/docs/contributing/documentation.mdx new file mode 100644 index 0000000000000..f3d6759f3185b --- /dev/null +++ b/docs/docs/contributing/documentation.mdx @@ -0,0 +1,46 @@ +--- +sidebar_label: Documentation +sidebar_position: 1 +--- +# Contribute Documentation + +The docs directory contains Documentation and API Reference. + +Documentation is built using [Docusaurus 2](https://docusaurus.io/). + +API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code. +For that reason, we ask that you add good documentation to all classes and methods. + +Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed. + +### Build Documentation Locally + +In the following commands, the prefix `api_` indicates that those are operations for the API Reference. + +Before building the documentation, it is always a good idea to clean the build directory: + +```bash +make docs_clean +make api_docs_clean +``` + +Next, you can build the documentation as outlined below: + +```bash +make docs_build +make api_docs_build +``` + +Finally, run the link checker to ensure all links are valid: + +```bash +make docs_linkcheck +make api_docs_linkcheck +``` + +### Verify Documentation changes + +After pushing documentation changes to the repository, you can preview and verify that the changes are +what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page. +This will take you to a preview of the documentation changes. +This preview is created by [Vercel](https://vercel.com/docs/getting-started-with-vercel). \ No newline at end of file diff --git a/docs/docs/contributing/index.mdx b/docs/docs/contributing/index.mdx new file mode 100644 index 0000000000000..f036deda0c13a --- /dev/null +++ b/docs/docs/contributing/index.mdx @@ -0,0 +1,44 @@ +--- +sidebar_position: 0 +--- +# Welcome Contributors + +Hi there! Thank you for even being interested in contributing to LangChain. +As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes. + +## 🗺️ Guidelines + +### 👩‍💻 Ways to contribute + +There are many ways to contribute to LangChain. Here are some common ways people contribute: + +- [**Documentation**](./documentation): Help improve our docs, including this one! +- [**Code**](./code): Help us write code, fix bugs, or improve our infrastructure. +- [**Integrations**](./integration): Help us integrate with your favorite vendors and tools. + +### + +### 🚩GitHub Issues + +Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests. + +There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues. + +If you start working on an issue, please assign it to yourself. + +If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature. +If two issues are related, or blocking, please link them rather than combining them. + +We will try to keep these issues as up-to-date as possible, though +with the rapid rate of development in this field some may get out of date. +If you notice this happening, please let us know. + +### 🙋Getting Help + +Our goal is to have the simplest developer setup possible. Should you experience any difficulty getting setup, please +contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is +smooth for future contributors. + +In a similar vein, we do enforce certain linting, formatting, and documentation standards in the codebase. +If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help - +we do not want these to get in the way of getting good code into the codebase. diff --git a/docs/docs/contributing/integrations.mdx b/docs/docs/contributing/integrations.mdx new file mode 100644 index 0000000000000..ab504b8f4a7b3 --- /dev/null +++ b/docs/docs/contributing/integrations.mdx @@ -0,0 +1,6 @@ +--- +sidebar_label: Integration +sidebar_position: 3 +--- +# Contribute Integrations + diff --git a/docs/docs/contributing/reference/_category_.yml b/docs/docs/contributing/reference/_category_.yml new file mode 100644 index 0000000000000..4d465c3be7a2d --- /dev/null +++ b/docs/docs/contributing/reference/_category_.yml @@ -0,0 +1,3 @@ +label: 'Reference' +collapsed: false + diff --git a/docs/docs/contributing/reference/packages.mdx b/docs/docs/contributing/reference/packages.mdx new file mode 100644 index 0000000000000..b1670734255f8 --- /dev/null +++ b/docs/docs/contributing/reference/packages.mdx @@ -0,0 +1,49 @@ +# Packages + +## 📕 Releases & Versioning + +As of now, LangChain has an ad hoc release process: releases are cut with high frequency by +a maintainer and published to [PyPI](https://pypi.org/). +The different packages are versioned slightly differently. + +### `langchain-core` + +`langchain-core` is currently on version `0.1.x`. + +As `langchain-core` contains the base abstractions and runtime for the whole LangChain ecosystem, we will communicate any breaking changes with advance notice and version bumps. The exception for this is anything in `langchain_core.beta`. The reason for `langchain_core.beta` is that given the rate of change of the field, being able to move quickly is still a priority, and this module is our attempt to do so. + +Minor version increases will occur for: + +- Breaking changes for any public interfaces NOT in `langchain_core.beta` + +Patch version increases will occur for: + +- Bug fixes +- New features +- Any changes to private interfaces +- Any changes to `langchain_core.beta` + +### `langchain` + +`langchain` is currently on version `0.0.x` + +All changes will be accompanied by a patch version increase. Any changes to public interfaces are nearly always done in a backwards compatible way and will be communicated ahead of time when they are not backwards compatible. + +We are targeting January 2024 for a release of `langchain` v0.1, at which point `langchain` will adopt the same versioning policy as `langchain-core`. + +### `langchain-community` + +`langchain-community` is currently on version `0.0.x` + +All changes will be accompanied by a patch version increase. + +### `langchain-experimental` + +`langchain-experimental` is currently on version `0.0.x` + +All changes will be accompanied by a patch version increase. + +## 🌟 Recognition + +If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)! +If you have a Twitter account you would like us to mention, please let us know in the PR or through another means. diff --git a/docs/sidebars.js b/docs/sidebars.js index 8468b3216c6ac..c0f6128d4ae55 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -135,4 +135,15 @@ module.exports = { link: { type: 'doc', id: "templates/index" } }, ], + contributing: [ + // { + // type: "category", + // label: "Contributing", + // items: [ + // { type: "autogenerated", dirName: "contributing" }, + // ], + // link: { type: 'doc', id: "contributing/index" } + // }, + {type: "autogenerated", dirName: "contributing" } + ], }; diff --git a/docs/vercel_build.sh b/docs/vercel_build.sh index 145a7e9acbc62..ec7b7c63a564d 100755 --- a/docs/vercel_build.sh +++ b/docs/vercel_build.sh @@ -19,6 +19,5 @@ mkdir docs/templates cp ../templates/docs/INDEX.md docs/templates/index.md python3.8 scripts/copy_templates.py cp ../cookbook/README.md src/pages/cookbook.mdx -cp ../.github/CONTRIBUTING.md docs/contributing.md wget -q https://raw.githubusercontent.com/langchain-ai/langserve/main/README.md -O docs/langserve.md quarto render docs/ From d681b919309019006638125e87bbace672dafa68 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 12:37:01 -0800 Subject: [PATCH 02/13] pages --- docs/docs/contributing/integrations.mdx | 125 ++++++++++++++++++ docs/docs/contributing/reference/packages.mdx | 4 + 2 files changed, 129 insertions(+) diff --git a/docs/docs/contributing/integrations.mdx b/docs/docs/contributing/integrations.mdx index ab504b8f4a7b3..79cf08cf85370 100644 --- a/docs/docs/contributing/integrations.mdx +++ b/docs/docs/contributing/integrations.mdx @@ -4,3 +4,128 @@ sidebar_position: 3 --- # Contribute Integrations +To begin, make sure you have all the dependencies outlined in guide on [Contributing Code](./code). + +There are a few different places you can contribute integrations for LangChain: + +- **Community**: For lighter-weight integrations that are primarily maintained by LangChain and the Open Source Community. +- **Partner Packages**: For independent packages that are co-maintained by LangChain and a partner. + +For the most part, new integrations should be added to the Community package. Partner packages require more maintenance as separate packages, so please confirm with the LangChain team before creating a new partner package. + +In the following sections, we'll walk through how to contribute to each of these packages from a fake company, `Parrot Link AI`. + +## Community Package + +The `langchain-community` package is in `libs/community` and contains most integrations. + +It is installed by users with `pip install langchain-community`, and exported members can be imported with code like + +```python +from langchain_community.chat_models import ParrotLinkLLM +from langchain_community.llms import ChatParrotLink +from langchain_community.vectorstores import ParrotLinkVectorStore +``` + +The community package relies on manually-installed dependent packages, so you will see errors if you try to import a package that is not installed. In our fake example, if you tried to import `ParrotLinkLLM` without installing `parrot-link-sdk`, you will see an `ImportError` telling you to install it when trying to use it. + +Let's say we wanted to implement a chat model for Parrot Link AI. We would create a new file in `libs/community/langchain_community/chat_models/parrot_link.py` with the following code: + +```python +from langchain_core.language_models.chat_models import BaseChatModel + +class ChatParrotLink(BaseChatModel): + """ChatParrotLink chat model. + + Example: + .. code-block:: python + + from langchain_parrot_link import ChatParrotLink + + model = ChatParrotLink() + """ + + ... +``` + +And we would write tests in: + +- Unit tests: `libs/community/tests/unit_tests/chat_models/test_parrot_link.py` +- Integration tests: `libs/community/tests/integration_tests/chat_models/test_parrot_link.py` + +And add documentation to: +- `docs/docs/integrations/chat/parrot_link.ipynb` + +- `docs/docs/ +## Partner Packages + +Partner packages are in `libs/partners/*` and are installed by users with `pip install langchain-{partner}`, and exported members can be imported with code like + +```python +from langchain_{partner} import X +``` + +### Set up a new package + +To set up a new partner package, use the latest version of the LangChain CLI. You can install or update it with: + +```bash +pip install -U langchain-cli +``` + +Let's say you want to create a new partner package working for a company called Parrot Link AI. + +Then, run the following command to create a new partner package: + +```bash +cd libs/partners +langchain-cli integration new +> Name: parrot-link +> Name of integration in PascalCase [ParrotLink]: ParrotLink +``` + +This will create a new package in `libs/partners/parrot-link` with the following structure: + +``` +libs/partners/parrot-link/ + langchain_parrot_link/ # folder containing your package + ... + tests/ + ... + docs/ # bootstrapped docs notebooks, must be moved to /docs in monorepo root + ... + scripts/ # scripts for CI + ... + LICENSE + README.md # fill out with information about your package + Makefile # default commands for CI + pyproject.toml # package metadata, mostly managed by Poetry + poetry.lock # package lockfile, managed by Poetry + .gitignore +``` + +### Implement your package + +First, add any dependencies your package needs, such as your company's SDK: + +```bash +poetry add parrot-link-sdk +``` + +If you need separate dependencies for type checking, you can add them to the `typing` group with: + +```bash +poetry add --group typing types-parrot-link-sdk +``` + +Then, implement your package in `libs/partners/parrot-link/langchain_parrot_link`. + +By default, this will include stubs for a Chat Model, an LLM, and/or a Vector Store. You should delete any of the files you won't use and remove them from `__init__.py`. + +### Write Unit and Integration Tests + +Some basic tests are generated in the tests/ directory. You should add more tests to cover your package's functionality. + +### Write documentation + +Documentation is generated from Jupyter notebooks in the `docs/` directory. You should move the generated notebooks to the relevant `docs/docs/integrations` directory in the monorepo root. \ No newline at end of file diff --git a/docs/docs/contributing/reference/packages.mdx b/docs/docs/contributing/reference/packages.mdx index b1670734255f8..5c06521a94f64 100644 --- a/docs/docs/contributing/reference/packages.mdx +++ b/docs/docs/contributing/reference/packages.mdx @@ -43,6 +43,10 @@ All changes will be accompanied by a patch version increase. All changes will be accompanied by a patch version increase. +### Partner Packages + +Partner packages are versioned independently. + ## 🌟 Recognition If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)! From 85824d0d17280b06a63cfb2a2ff2f6e29f671fa5 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 12:51:35 -0800 Subject: [PATCH 03/13] steps --- docs/docs/contributing/integrations.mdx | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/docs/contributing/integrations.mdx b/docs/docs/contributing/integrations.mdx index 79cf08cf85370..65763b50775d3 100644 --- a/docs/docs/contributing/integrations.mdx +++ b/docs/docs/contributing/integrations.mdx @@ -128,4 +128,20 @@ Some basic tests are generated in the tests/ directory. You should add more test ### Write documentation -Documentation is generated from Jupyter notebooks in the `docs/` directory. You should move the generated notebooks to the relevant `docs/docs/integrations` directory in the monorepo root. \ No newline at end of file +Documentation is generated from Jupyter notebooks in the `docs/` directory. You should move the generated notebooks to the relevant `docs/docs/integrations` directory in the monorepo root. + +### Additional steps + +Contributor steps: + +- [ ] Add the new package to the API reference dropdown in `docs/api_reference/themes/scikit-learn-modern/nav.html` +- [ ] Add package (e.g. `langchain-parrot-link`) to API docs build in `docs/api_reference/requirements.txt` +- [ ] Add secret names to manual integrations workflow in `.github/workflows/_integration_test.yml` +- [ ] Add secrets to release workflow (for pre-release testing) in `.github/workflows/_release.yml` +- [ ] Add library choice to top of `.github/workflows/_release.yml` + +Maintainer steps (Contributors should **not** do these): + +- [ ] set up pypi and test pypi projects +- [ ] add credential secrets to Github Actions +- [ ] add package to conda-forge From 84dad2e5eb3da87daf76ad28a6d8827202eef932 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 12:56:28 -0800 Subject: [PATCH 04/13] slightly more --- .github/CONTRIBUTING.md | 39 ++++++++++++++++++++++++++++++++ docs/docs/contributing/index.mdx | 2 -- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index bab7857998d85..caa859e6c8c79 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -3,4 +3,43 @@ Hi there! Thank you for even being interested in contributing to LangChain. As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes. +To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/) + +## 🗺️ Guidelines + +### 👩‍💻 Ways to contribute + +There are many ways to contribute to LangChain. Here are some common ways people contribute: + +- [**Documentation**](https://python.langchain.com/docs/contributing/documentation): Help improve our docs, including this one! +- [**Code**](https://python.langchain.com/docs/contributing/code): Help us write code, fix bugs, or improve our infrastructure. +- [**Integrations**](https://python.langchain.com/docs/contributing/integration): Help us integrate with your favorite vendors and tools. + +### 🚩GitHub Issues + +Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests. + +There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues. + +If you start working on an issue, please assign it to yourself. + +If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature. +If two issues are related, or blocking, please link them rather than combining them. + +We will try to keep these issues as up-to-date as possible, though +with the rapid rate of development in this field some may get out of date. +If you notice this happening, please let us know. + +### 🙋Getting Help + +Our goal is to have the simplest developer setup possible. Should you experience any difficulty getting setup, please +contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is +smooth for future contributors. + +In a similar vein, we do enforce certain linting, formatting, and documentation standards in the codebase. +If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help - +we do not want these to get in the way of getting good code into the codebase. + +### Contributor Documentation + To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/) \ No newline at end of file diff --git a/docs/docs/contributing/index.mdx b/docs/docs/contributing/index.mdx index f036deda0c13a..2200e835d6bd3 100644 --- a/docs/docs/contributing/index.mdx +++ b/docs/docs/contributing/index.mdx @@ -16,8 +16,6 @@ There are many ways to contribute to LangChain. Here are some common ways people - [**Code**](./code): Help us write code, fix bugs, or improve our infrastructure. - [**Integrations**](./integration): Help us integrate with your favorite vendors and tools. -### - ### 🚩GitHub Issues Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests. From 897a924cb8f05a6338baeddc6647528a6645be56 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 13:56:10 -0800 Subject: [PATCH 05/13] refs --- .github/ISSUE_TEMPLATE/feature-request.yml | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- README.md | 2 +- docs/docs/community.md | 2 +- docs/docs/contributing/code.mdx | 30 ++-- docs/docs/contributing/documentation.mdx | 30 +++- docs/docs/contributing/reference/packages.mdx | 18 ++- docs/docs/contributing/reference/testing.mdx | 143 ++++++++++++++++++ libs/community/README.md | 2 +- libs/community/pyproject.toml | 4 +- libs/core/README.md | 2 +- libs/langchain/README.md | 2 +- libs/langchain/pyproject.toml | 4 +- libs/langchain/tests/README.md | 124 +-------------- 14 files changed, 201 insertions(+), 166 deletions(-) create mode 100644 docs/docs/contributing/reference/testing.mdx diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index f11292793b9f1..3b87dcd4046ea 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -27,4 +27,4 @@ body: attributes: label: Your contribution description: | - Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md) + Is there any way that you could help, e.g. by submitting a PR? Make sure to read the [Contributing Guide](https://python.langchain.com/docs/contributing/) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 052839c260428..a32336a3c165f 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -10,7 +10,7 @@ Replace this entire comment with: Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: -https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md +https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, diff --git a/README.md b/README.md index 428e762164e75..6a0a91a6f2006 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ Please see [here](https://python.langchain.com) for full documentation, which in As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation. -For detailed information on how to contribute, see [here](.github/CONTRIBUTING.md). +For detailed information on how to contribute, see [here](https://python.langchain.com/docs/contributing/). ## 🌟 Contributors diff --git a/docs/docs/community.md b/docs/docs/community.md index c4d24d45b68e3..81749dffafde7 100644 --- a/docs/docs/community.md +++ b/docs/docs/community.md @@ -18,7 +18,7 @@ Whether you’re new to LangChain, looking to go deeper, or just want to get mor LangChain is the product of over 5,000+ contributions by 1,500+ contributors, and there is ******still****** so much to do together. Here are some ways to get involved: - **[Open a pull request](https://github.com/langchain-ai/langchain/issues):** We’d appreciate all forms of contributions–new features, infrastructure improvements, better documentation, bug fixes, etc. If you have an improvement or an idea, we’d love to work on it with you. -- **[Read our contributor guidelines:](https://github.com/langchain-ai/langchain/blob/bbd22b9b761389a5e40fc45b0570e1830aabb707/.github/CONTRIBUTING.md)** We ask contributors to follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow, run a few local checks for formatting, linting, and testing before submitting, and follow certain documentation and testing conventions. +- **[Read our contributor guidelines:](./contributing/)** We ask contributors to follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow, run a few local checks for formatting, linting, and testing before submitting, and follow certain documentation and testing conventions. - **First time contributor?** [Try one of these PRs with the “good first issue” tag](https://github.com/langchain-ai/langchain/contribute). - **Become an expert:** Our experts help the community by answering product questions in Discord. If that’s a role you’d like to play, we’d be so grateful! (And we have some special experts-only goodies/perks we can tell you more about). Send us an email to introduce yourself at hello@langchain.dev and we’ll take it from there! - **Integrate with LangChain:** If your product integrates with LangChain–or aspires to–we want to help make sure the experience is as smooth as possible for you and end users. Send us an email at hello@langchain.dev and tell us what you’re working on. diff --git a/docs/docs/contributing/code.mdx b/docs/docs/contributing/code.mdx index c20a0772b2a59..4ac9532e0ebfe 100644 --- a/docs/docs/contributing/code.mdx +++ b/docs/docs/contributing/code.mdx @@ -49,22 +49,23 @@ This repository contains multiple packages: - `langchain-community`: Third-party integrations of various components. - `langchain`: Chains, agents, and retrieval logic that makes up the cognitive architecture of your applications. - `langchain-experimental`: Components and chains that are experimental, either in the sense that the techniques are novel and still being tested, or they require giving the LLM more access than would be possible in most production systems. +- Partner integrations: Partner packages in `libs/partners` that are independently verison controlled. Each of these has its own development environment. Docs are run from the top-level makefile, but development is split across separate test & release flows. -For this quickstart, start with langchain: +For this quickstart, start with langchain-community: ```bash -cd libs/langchain +cd libs/community ``` ### Local Development Dependencies -Install langchain development requirements (for running langchain, running examples, linting, formatting, tests, and coverage): +Install langchain-community development requirements (for running langchain, running examples, linting, formatting, tests, and coverage): ```bash -poetry install --with test +poetry install --with lint,typing,test,integration_tests ``` Then verify dependency installation: @@ -73,8 +74,6 @@ Then verify dependency installation: make test ``` -If the tests don't pass, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`. - If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running Poetry v1.6.1+. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases. If you are still seeing this bug on v1.6.1, you may also try disabling "modern installation" @@ -83,7 +82,7 @@ See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for ### Testing -_some test dependencies are optional; see section about optional dependencies_. +_In `langchain`, `langchain-community`, and `langchain-experimental`, some test dependencies are optional; see section about optional dependencies_. Unit tests cover modular logic that does not require calls to outside APIs. If you add new logic, please add a unit test. @@ -100,7 +99,7 @@ To run unit tests in Docker: make docker_tests ``` -There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available. +There are also [integration tests and code-coverage](./reference/testing) available. ### Only develop langchain_core or langchain_experimental @@ -204,7 +203,9 @@ ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogy ## Working with Optional Dependencies -Langchain relies heavily on optional dependencies to keep the Langchain package lightweight. +`langchain`, `langchain-community`, and `langchain-experimental` rely on optional dependencies to keep these packages lightweight. + +`langchain-core` and partner packages **do not use** optional dependencies in this way. You only need to add a new dependency if a **unit test** relies on the package. If your package is only required for **integration tests**, then you can skip these @@ -248,14 +249,3 @@ poetry run jupyter notebook ``` When you run `poetry install`, the `langchain` package is installed as editable in the virtualenv, so your new logic can be imported into the notebook. - -## Documentation - -While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing. -This covers how to get started contributing to documentation. - -From the top-level of this repo, install documentation dependencies: - -```bash -poetry install -``` \ No newline at end of file diff --git a/docs/docs/contributing/documentation.mdx b/docs/docs/contributing/documentation.mdx index f3d6759f3185b..bb2351a834c0f 100644 --- a/docs/docs/contributing/documentation.mdx +++ b/docs/docs/contributing/documentation.mdx @@ -6,14 +6,21 @@ sidebar_position: 1 The docs directory contains Documentation and API Reference. -Documentation is built using [Docusaurus 2](https://docusaurus.io/). +Documentation is built using [Quarto](https://quarto.org) and [Docusaurus 2](https://docusaurus.io/). -API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code. +API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code and are hosted by [Read the Docs](https://readthedocs.org/). For that reason, we ask that you add good documentation to all classes and methods. Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed. -### Build Documentation Locally +## Build Documentation Locally + +### Install dependencies + +- [Quarto](https://quarto.org) - package that converts Jupyter notebooks (`.ipynb` files) into mdx files for serving in Docusaurus. +- `poetry install` from the monorepo root + +### Building In the following commands, the prefix `api_` indicates that those are operations for the API Reference. @@ -38,7 +45,22 @@ make docs_linkcheck make api_docs_linkcheck ``` -### Verify Documentation changes +### Linting and Formatting + +The docs are linted from the monorepo root. To lint the docs, run the following from there: + +```bash +poetry install --with lint,typing +make lint +``` + +If you have formatting-related errors, you can fix them automatically with: + +```bash +make format +``` + +## Verify Documentation changes After pushing documentation changes to the repository, you can preview and verify that the changes are what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page. diff --git a/docs/docs/contributing/reference/packages.mdx b/docs/docs/contributing/reference/packages.mdx index 5c06521a94f64..9f7c2f39eab89 100644 --- a/docs/docs/contributing/reference/packages.mdx +++ b/docs/docs/contributing/reference/packages.mdx @@ -1,12 +1,14 @@ -# Packages +--- +sidebar_label: Package Versioning +--- -## 📕 Releases & Versioning +# 📕 Package Versioning As of now, LangChain has an ad hoc release process: releases are cut with high frequency by a maintainer and published to [PyPI](https://pypi.org/). The different packages are versioned slightly differently. -### `langchain-core` +## `langchain-core` `langchain-core` is currently on version `0.1.x`. @@ -23,7 +25,7 @@ Patch version increases will occur for: - Any changes to private interfaces - Any changes to `langchain_core.beta` -### `langchain` +## `langchain` `langchain` is currently on version `0.0.x` @@ -31,23 +33,23 @@ All changes will be accompanied by a patch version increase. Any changes to publ We are targeting January 2024 for a release of `langchain` v0.1, at which point `langchain` will adopt the same versioning policy as `langchain-core`. -### `langchain-community` +## `langchain-community` `langchain-community` is currently on version `0.0.x` All changes will be accompanied by a patch version increase. -### `langchain-experimental` +## `langchain-experimental` `langchain-experimental` is currently on version `0.0.x` All changes will be accompanied by a patch version increase. -### Partner Packages +## Partner Packages Partner packages are versioned independently. -## 🌟 Recognition +# 🌟 Recognition If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)! If you have a Twitter account you would like us to mention, please let us know in the PR or through another means. diff --git a/docs/docs/contributing/reference/testing.mdx b/docs/docs/contributing/reference/testing.mdx new file mode 100644 index 0000000000000..89715093e978d --- /dev/null +++ b/docs/docs/contributing/reference/testing.mdx @@ -0,0 +1,143 @@ +# Testing + +All of our packages have unit tests and integration tests, and we favor unit tests over integration tests. + +Unit tests run on every pull request, so they should be fast and reliable. + +Integration tests run once a day, and they require more setup, so they should be reserved for confirming interface points with external services. + +## Unit Tests + +Unit tests cover modular logic that does not require calls to outside APIs. +If you add new logic, please add a unit test. + +To install dependencies for unit tests: + +```bash +poetry install --with test +``` + +To run unit tests: + +```bash +make test +``` + +To run unit tests in Docker: + +```bash +make docker_tests +``` + +To run a specific test: + +```bash +TEST_FILE=tests/unit_tests/test_imports.py make test +``` + +## Integration Tests + +Integration tests cover logic that requires making calls to outside APIs (often integration with other services). +If you add support for a new external API, please add a new integration test. + +**Warning:** Almost no tests should be integration tests. + + Tests that require making network connections make it difficult for other + developers to test the code. + + Instead favor relying on `responses` library and/or mock.patch to mock + requests using small fixtures. + +To install dependencies for integration tests: + +```bash +poetry install --with test,test_integration +``` + +To run integration tests: + +```bash +make integration_tests +``` + +### Prepare + +The integration tests use several search engines and databases. The tests +aim to verify the correct behavior of the engines and databases according to +their specifications and requirements. + +To run some integration tests, such as tests located in +`tests/integration_tests/vectorstores/`, you will need to install the following +software: + +- Docker +- Python 3.8.1 or later + +Any new dependencies should be added by running: + +```bash +# add package and install it after adding: +poetry add tiktoken@latest --group "test_integration" && poetry install --with test_integration +``` + +Before running any tests, you should start a specific Docker container that has all the +necessary dependencies installed. For instance, we use the `elasticsearch.yml` container +for `test_elasticsearch.py`: + +```bash +cd tests/integration_tests/vectorstores/docker-compose +docker-compose -f elasticsearch.yml up +``` + +For environments that requires more involving preparation, look for `*.sh`. For instance, +`opensearch.sh` builds a required docker image and then launch opensearch. + + +### Prepare environment variables for local testing: + +- copy `tests/integration_tests/.env.example` to `tests/integration_tests/.env` +- set variables in `tests/integration_tests/.env` file, e.g `OPENAI_API_KEY` + +Additionally, it's important to note that some integration tests may require certain +environment variables to be set, such as `OPENAI_API_KEY`. Be sure to set any required +environment variables before running the tests to ensure they run correctly. + +### Recording HTTP interactions with pytest-vcr + +Some of the integration tests in this repository involve making HTTP requests to +external services. To prevent these requests from being made every time the tests are +run, we use pytest-vcr to record and replay HTTP interactions. + +When running tests in a CI/CD pipeline, you may not want to modify the existing +cassettes. You can use the --vcr-record=none command-line option to disable recording +new cassettes. Here's an example: + +```bash +pytest --log-cli-level=10 tests/integration_tests/vectorstores/test_pinecone.py --vcr-record=none +pytest tests/integration_tests/vectorstores/test_elasticsearch.py --vcr-record=none + +``` + +### Run some tests with coverage: + +```bash +pytest tests/integration_tests/vectorstores/test_elasticsearch.py --cov=langchain --cov-report=html +start "" htmlcov/index.html || open htmlcov/index.html + +``` + +## Coverage + +Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle. + +Coverage requires the dependencies for integration tests: + +```bash +poetry install --with test_integration +``` + +To get a report of current coverage, run the following: + +```bash +make coverage +``` diff --git a/libs/community/README.md b/libs/community/README.md index c1116994d9e36..07c155beab0fa 100644 --- a/libs/community/README.md +++ b/libs/community/README.md @@ -27,4 +27,4 @@ All changes will be accompanied by a patch version increase. As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation. -For detailed information on how to contribute, see [here](../../.github/CONTRIBUTING.md). \ No newline at end of file +For detailed information on how to contribute, see the [Contributing Guide](https://python.langchain.com/docs/contributing/). \ No newline at end of file diff --git a/libs/community/pyproject.toml b/libs/community/pyproject.toml index 2b2868005fdf1..7b03c90ee3223 100644 --- a/libs/community/pyproject.toml +++ b/libs/community/pyproject.toml @@ -130,8 +130,8 @@ optional = true # developers from being able to easily run them. # Instead write unit tests that use the `responses` library or mock.patch with # fixtures. Keep the fixtures minimal. -# See CONTRIBUTING.md for more instructions on working with optional dependencies. -# https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md#working-with-optional-dependencies +# See Contributing Guide for more instructions on working with optional dependencies. +# https://python.langchain.com/docs/contributing/code#working-with-optional-dependencies pytest-vcr = "^1.0.2" wrapt = "^1.15.0" openai = "^1" diff --git a/libs/core/README.md b/libs/core/README.md index d7026461a81c9..0642c8fb8766d 100644 --- a/libs/core/README.md +++ b/libs/core/README.md @@ -55,4 +55,4 @@ Patch version increases will occur for: As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation. -For detailed information on how to contribute, see [here](../../.github/CONTRIBUTING.md). \ No newline at end of file +For detailed information on how to contribute, see the [Contributing Guide](https://python.langchain.com/docs/contributing/). \ No newline at end of file diff --git a/libs/langchain/README.md b/libs/langchain/README.md index f90965397e72e..2c8d69bcb4919 100644 --- a/libs/langchain/README.md +++ b/libs/langchain/README.md @@ -93,4 +93,4 @@ For more information on these concepts, please see our [full documentation](http As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation. -For detailed information on how to contribute, see [here](../../.github/CONTRIBUTING.md). +For detailed information on how to contribute, see the [Contributing Guide](https://python.langchain.com/docs/contributing/). diff --git a/libs/langchain/pyproject.toml b/libs/langchain/pyproject.toml index 6dc4655314c02..ed6116afc9fa6 100644 --- a/libs/langchain/pyproject.toml +++ b/libs/langchain/pyproject.toml @@ -157,8 +157,8 @@ optional = true # developers from being able to easily run them. # Instead write unit tests that use the `responses` library or mock.patch with # fixtures. Keep the fixtures minimal. -# See CONTRIBUTING.md for more instructions on working with optional dependencies. -# https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md#working-with-optional-dependencies +# See the Contributing Guide for more instructions on working with optional dependencies. +# https://python.langchain.com/docs/contributing/code#working-with-optional-dependencies pytest-vcr = "^1.0.2" wrapt = "^1.15.0" openai = "^1" diff --git a/libs/langchain/tests/README.md b/libs/langchain/tests/README.md index 6190d969f48ca..a825ff3ca09e6 100644 --- a/libs/langchain/tests/README.md +++ b/libs/langchain/tests/README.md @@ -1,125 +1,3 @@ # Langchain Tests -## Unit Tests - -Unit tests cover modular logic that does not require calls to outside APIs. -If you add new logic, please add a unit test. - -To run unit tests: - -```bash -make test -``` - -To run unit tests in Docker: - -```bash -make docker_tests -``` - -## Integration Tests - -Integration tests cover logic that requires making calls to outside APIs (often integration with other services). -If you add support for a new external API, please add a new integration test. - -**warning** Almost no tests should be integration tests. - - Tests that require making network connections make it difficult for other - developers to test the code. - - Instead favor relying on `responses` library and/or mock.patch to mock - requests using small fixtures. - -To install dependencies for integration tests: - -```bash -poetry install --with test_integration -``` - -To run integration tests: - -```bash -make integration_tests -``` - -### Prepare - -The integration tests exercise several search engines and databases. The tests -aim to verify the correct behavior of the engines and databases according to -their specifications and requirements. - -To run some integration tests, such as tests located in -`tests/integration_tests/vectorstores/`, you will need to install the following -software: - -- Docker -- Python 3.8.1 or later - -Any new dependencies should be added by running: - -```bash -# add package and install it after adding: -poetry add tiktoken@latest --group "test_integration" && poetry install --with test_integration -``` - -Before running any tests, you should start a specific Docker container that has all the -necessary dependencies installed. For instance, we use the `elasticsearch.yml` container -for `test_elasticsearch.py`: - -```bash -cd tests/integration_tests/vectorstores/docker-compose -docker-compose -f elasticsearch.yml up -``` - -For environments that requires more involving preparation, look for `*.sh`. For instance, -`opensearch.sh` builds a required docker image and then launch opensearch. - - -### Prepare environment variables for local testing: - -- copy `tests/integration_tests/.env.example` to `tests/integration_tests/.env` -- set variables in `tests/integration_tests/.env` file, e.g `OPENAI_API_KEY` - -Additionally, it's important to note that some integration tests may require certain -environment variables to be set, such as `OPENAI_API_KEY`. Be sure to set any required -environment variables before running the tests to ensure they run correctly. - -### Recording HTTP interactions with pytest-vcr - -Some of the integration tests in this repository involve making HTTP requests to -external services. To prevent these requests from being made every time the tests are -run, we use pytest-vcr to record and replay HTTP interactions. - -When running tests in a CI/CD pipeline, you may not want to modify the existing -cassettes. You can use the --vcr-record=none command-line option to disable recording -new cassettes. Here's an example: - -```bash -pytest --log-cli-level=10 tests/integration_tests/vectorstores/test_pinecone.py --vcr-record=none -pytest tests/integration_tests/vectorstores/test_elasticsearch.py --vcr-record=none - -``` - -### Run some tests with coverage: - -```bash -pytest tests/integration_tests/vectorstores/test_elasticsearch.py --cov=langchain --cov-report=html -start "" htmlcov/index.html || open htmlcov/index.html - -``` - -## Coverage - -Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle. - -Coverage requires the dependencies for integration tests: - -```bash -poetry install --with test_integration -``` - -To get a report of current coverage, run the following: - -```bash -make coverage -``` +[This guide has moved to the docs](https://python.langchain.com/docs/contributing/reference/testing) From 01f7b1a688487c810b7c533ad5ba08189a79f4c0 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 14:00:53 -0800 Subject: [PATCH 06/13] link --- docs/docs/contributing/integrations.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/docs/contributing/integrations.mdx b/docs/docs/contributing/integrations.mdx index 65763b50775d3..9273bf0c15f5f 100644 --- a/docs/docs/contributing/integrations.mdx +++ b/docs/docs/contributing/integrations.mdx @@ -126,6 +126,8 @@ By default, this will include stubs for a Chat Model, an LLM, and/or a Vector St Some basic tests are generated in the tests/ directory. You should add more tests to cover your package's functionality. +For information on running and implementing tests, see the [Testing guide](./reference/testing). + ### Write documentation Documentation is generated from Jupyter notebooks in the `docs/` directory. You should move the generated notebooks to the relevant `docs/docs/integrations` directory in the monorepo root. From 991dddd32b8b3a9d152bd4201d85cc3f0eed0780 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 14:11:20 -0800 Subject: [PATCH 07/13] updated docs readme --- docs/README.md | 50 ++------------------------------------------------ 1 file changed, 2 insertions(+), 48 deletions(-) diff --git a/docs/README.md b/docs/README.md index 3e8dfe2e5d653..7593716a9eacd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,49 +1,3 @@ -# Website +# LangChain Documentation -This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator. - -### Installation - -``` -$ yarn -``` - -### Local Development - -``` -$ yarn start -``` - -This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. - -### Build - -``` -$ yarn build -``` - -This command generates static content into the `build` directory and can be served using any static contents hosting service. - -### Deployment - -Using SSH: - -``` -$ USE_SSH=true yarn deploy -``` - -Not using SSH: - -``` -$ GIT_USER= yarn deploy -``` - -If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch. - -### Continuous Integration - -Some common defaults for linting/formatting have been set for you. If you integrate your project with an open-source Continuous Integration system (e.g. Travis CI, CircleCI), you may check for issues using the following command. - -``` -$ yarn ci -``` +For more information on contributing to our documentation, see the [Documentation Contributing Guide](https://python.langchain.com/docs/contributing/documentation) From 518de03efc3f58695042e2d7200cc3fd36d14a49 Mon Sep 17 00:00:00 2001 From: Erick Friis Date: Fri, 15 Dec 2023 14:11:48 -0800 Subject: [PATCH 08/13] attempt build from conf var --- docs/api_reference/conf.py | 9 ++++++--- docs/api_reference/themes/scikit-learn-modern/nav.html | 3 --- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/api_reference/conf.py b/docs/api_reference/conf.py index 9795e9dbf6378..58963e334629c 100644 --- a/docs/api_reference/conf.py +++ b/docs/api_reference/conf.py @@ -72,8 +72,8 @@ def setup(app): # -- Project information ----------------------------------------------------- project = "🦜🔗 LangChain" -copyright = "2023, Harrison Chase" -author = "Harrison Chase" +copyright = "2023, LangChain, Inc." +author = "LangChain, Inc." version = data["tool"]["poetry"]["version"] release = version @@ -143,11 +143,14 @@ def setup(app): html_context = { "display_github": True, # Integrate GitHub - "github_user": "hwchase17", # Username + "github_user": "langchain-ai", # Username "github_repo": "langchain", # Repo name "github_version": "master", # Version "conf_py_path": "/docs/api_reference", # Path in the checkout to the docs root "redirects": redirects, + "drop_down_navigation": [ + ("Google Generative AI", "https://python.langchain.com", "") + ], } # Add any paths that contain custom static files (such as style sheets) here, diff --git a/docs/api_reference/themes/scikit-learn-modern/nav.html b/docs/api_reference/themes/scikit-learn-modern/nav.html index da3fdae96b3cf..dd14c4c106645 100644 --- a/docs/api_reference/themes/scikit-learn-modern/nav.html +++ b/docs/api_reference/themes/scikit-learn-modern/nav.html @@ -7,9 +7,6 @@ {%- endif %} {# title, link, link_attrs #} -{%- set drop_down_navigation = [ - ('Google Generative AI', pathto('google_genai_api_reference'), ''),] --%}