Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pgvector #162

Merged
merged 2 commits into from
Feb 8, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions integrations/pgvector-documentstore.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
layout: integration
name: pgvector
description: A Document Store for storing and retrieval from pgvector
authors:
- name: deepset
socials:
github: deepset-ai
twitter: deepset_ai
linkedin: deepset-ai
pypi: https://pypi.org/project/pgvector-haystack/
repo: https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector
type: Document Store
report_issue: https://github.com/deepset-ai/haystack-core-integrations/issues
version: Haystack 2.0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the logo is missing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pgvector does not have a logo. Maybe because it is an extension of PostgreSQL.
We could also insert the logo of PostgreSQL, but it would be a bit misleading

toc: true
---

[![PyPI - Version](https://img.shields.io/pypi/v/pgvector-haystack.svg)](https://pypi.org/project/pgvector-haystack/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pgvector-haystack.svg)](https://pypi.org/project/pgvector-haystack/)
[![test](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/pgvector.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/pgvector.yml)

-----

**Table of Contents**

- Pgvector Document Store for Haystack
- [Installation](#installation)
- [Usage](#usage)
- [Examples](#examples)
- [License](#license)

## Installation
pgvector is an extension for PostgreSQL that adds support for vector similarity search.

To quickly setup a PostgreSQL database with pgvector, you can use Docker:
```bash
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
```
anakin87 marked this conversation as resolved.
Show resolved Hide resolved

For more information on how to install pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).

Use `pip` to install `pgvector-haystack`:
```bash
pip install pgvector-haystack
```
## Usage
Once installed, initialize PgvectorDocumentStore:

```python
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore

document_store = PgvectorDocumentStore(
connection_string="postgresql://postgres:postgres@localhost:5432/postgres",
table_name="haystack_docs",
embedding_dimension=768,
vector_function="cosine_similarity",
recreate_table=True,
search_strategy="hnsw",
)
```

### Writing Documents to PgvectorDocumentStore
To write documents to `PgvectorDocumentStore`, create an indexing Pipeline.

```python
from haystack.components.converters import TextFileToDocument
from haystack.components.writers import DocumentWriter
from haystack.components.embedders import SentenceTransformersDocumentEmbedder

indexing = Pipeline()
indexing.add_component("converter", TextFileToDocument())
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("converter", "embedder")
indexing.connect("embedder", "writer")
indexing.run({"converter": {"sources": file_paths}})
```

### Retrieval from PgvectorDocumentStore
You can retrieve Documents similar to a given query using a simple Pipeline.

```python
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack_integrations.components.retrievers.pgvector import PgvectorEmbeddingRetriever
anakin87 marked this conversation as resolved.
Show resolved Hide resolved

querying = Pipeline()
querying.add_component("embedder", SentenceTransformersTextEmbedder())
querying.add_component("retriever", PgvectorEmbeddingRetriever(document_store=document_store, top_k=3))
querying.connect("embedder", "retriever")

results = querying.run({"embedder": {"text": "my query"}})
```

## Examples
You can find a code example showing how to use the Document Store and the Retriever under the `examples/` folder of [this repo](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector).

## License

`pgvector-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.