Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(qdrant): Support custom payload also when using fromExistingCollection #7069

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(qdrant): Support custom payload also when using fromExistingCollection #7069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 7 additions & 11 deletions libs/langchain-qdrant/src/tests/vectorstores.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,12 @@ test("QdrantVectorStore adds vectors with custom payload", async () => {
});

// Define a custom payload
const customPayload = {
customPayload: [
const customPayload = [
{
customField1: "value1",
customField2: "value2",
},
],
};
];

// Add documents with custom payload
await qdrantVectorStore.addDocuments(
Expand All @@ -83,7 +81,7 @@ test("QdrantVectorStore adds vectors with custom payload", async () => {
payload: expect.objectContaining({
content: "hello",
metadata: {},
customPayload: customPayload.customPayload[0],
customPayload: customPayload[0],
}),
}),
],
Expand All @@ -109,17 +107,15 @@ test("QdrantVectorStore adds vectors with multiple custom payload", async () =>
});

// Define a custom payload
const customPayload = {
customPayload: [
const customPayload = [
{
customField1: "value1",
customField2: "value2",
},
{
customField3: "value3",
},
],
};
];

// Add documents with custom payload
await qdrantVectorStore.addDocuments(
Expand Down Expand Up @@ -149,14 +145,14 @@ test("QdrantVectorStore adds vectors with multiple custom payload", async () =>
payload: expect.objectContaining({
content: "hello",
metadata: {},
customPayload: customPayload.customPayload[0],
customPayload: customPayload[0],
}),
}),
expect.objectContaining({
payload: expect.objectContaining({
content: "Goodbye",
metadata: {},
customPayload: customPayload.customPayload[1],
customPayload: customPayload[1],
}),
}),
expect.objectContaining({
Expand Down
60 changes: 33 additions & 27 deletions libs/langchain-qdrant/src/vectorstores.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,16 @@ export interface QdrantLibArgs {
apiKey?: string;
collectionName?: string;
collectionConfig?: QdrantSchemas["CreateCollection"];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
customPayload?: Record<string, any>[];
customPayload?: QdrantCustomPayload;
contentPayloadKey?: string;
metadataPayloadKey?: string;
}

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export type QdrantCustomPayload = Record<string, any>;

export type QdrantAddDocumentOptions = {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
customPayload: Record<string, any>[];
customPayload: QdrantCustomPayload[];
};

export type QdrantFilter = QdrantSchemas["Filter"];
Expand Down Expand Up @@ -78,6 +79,8 @@ export class QdrantVectorStore extends VectorStore {

metadataPayloadKey: string;

customPayload?: QdrantCustomPayload;

_vectorstoreType(): string {
return "qdrant";
}
Expand All @@ -103,6 +106,8 @@ export class QdrantVectorStore extends VectorStore {

this.collectionConfig = args.collectionConfig;

this.customPayload = args.customPayload;

this.contentPayloadKey = args.contentPayloadKey ?? CONTENT_KEY;

this.metadataPayloadKey = args.metadataPayloadKey ?? METADATA_KEY;
Expand All @@ -113,18 +118,18 @@ export class QdrantVectorStore extends VectorStore {
* from the documents using the `Embeddings` instance and then adds the
* vectors to the database.
* @param documents Array of `Document` instances to be added to the Qdrant database.
* @param documentOptions Optional `QdrantAddDocumentOptions` which has a list of JSON objects for extra querying
* @param customPayloads Optional `QdrantCustomPayload` array which has a list of JSON objects for extra querying.
* @returns Promise that resolves when the documents have been added to the database.
*/
async addDocuments(
documents: Document[],
documentOptions?: QdrantAddDocumentOptions
customPayloads?: QdrantCustomPayload[],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is breaking isn't it?

Would prefer to keep the wrapping

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, feel free to keep it 🫣 I understand the urge, but in my opinion I'm always for the correct way instead producing the legacy code of tomorrow. It's yet already complicated enough with the customPayloads on document level and the customPayload on dbConfig level.

But I understand that you don't want to have breaking changes, even a simple one like this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's a fine line - I'd err towards keeping it for now with a note to change it in the next minor bump.

): Promise<void> {
const texts = documents.map(({ pageContent }) => pageContent);
await this.addVectors(
await this.embeddings.embedDocuments(texts),
documents,
documentOptions
customPayloads
);
}

Expand All @@ -134,29 +139,35 @@ export class QdrantVectorStore extends VectorStore {
* database.
* @param vectors Array of vectors to be added to the Qdrant database.
* @param documents Array of `Document` instances associated with the vectors.
* @param documentOptions Optional `QdrantAddDocumentOptions` which has a list of JSON objects for extra querying
* @param customPayloads Optional `QdrantCustomPayload` array which has a list of JSON objects for extra querying. Will be merged with custom payload of dbConfig.
* @returns Promise that resolves when the vectors have been added to the database.
*/
async addVectors(
vectors: number[][],
documents: Document[],
documentOptions?: QdrantAddDocumentOptions
customPayloads?: QdrantCustomPayload[],
): Promise<void> {
if (vectors.length === 0) {
return;
}

await this.ensureCollection();

const points = vectors.map((embedding, idx) => ({
id: documents[idx].id ?? uuid(),
vector: embedding,
payload: {
[this.contentPayloadKey]: documents[idx].pageContent,
[this.metadataPayloadKey]: documents[idx].metadata,
customPayload: documentOptions?.customPayload[idx],
},
}));
const points = vectors.map((embedding, idx) => {
const customPayload = {
...this.customPayload,
...customPayloads?.[idx],
}
return {
id: documents[idx].id ?? uuid(),
vector: embedding,
payload: {
[this.contentPayloadKey]: documents[idx].pageContent,
[this.metadataPayloadKey]: documents[idx].metadata,
customPayload: Object.keys(customPayload).length > 0 ? customPayload : undefined,
},
}
});

try {
await this.client.upsert(this.collectionName, {
Expand Down Expand Up @@ -332,22 +343,17 @@ export class QdrantVectorStore extends VectorStore {
* @param docs Array of `Document` instances to be added to the Qdrant database.
* @param embeddings `Embeddings` instance used to generate vectors from the documents.
* @param dbConfig `QdrantLibArgs` instance specifying the configuration for the Qdrant database.
* @param customPayloads Optional `QdrantCustomPayload` array which has a list of JSON objects for extra querying.
* @returns Promise that resolves with a new `QdrantVectorStore` instance.
*/
static async fromDocuments(
docs: Document[],
embeddings: EmbeddingsInterface,
dbConfig: QdrantLibArgs
dbConfig: QdrantLibArgs,
customPayloads?: QdrantCustomPayload[]
): Promise<QdrantVectorStore> {
const instance = new this(embeddings, dbConfig);
if (dbConfig.customPayload) {
const documentOptions = {
customPayload: dbConfig?.customPayload,
};
await instance.addDocuments(docs, documentOptions);
} else {
await instance.addDocuments(docs);
}
await instance.addDocuments(docs, customPayloads);
return instance;
}

Expand Down