ValueError:Batch size 1046409 exceeds maximum batch size 41666

[yingxin-chen]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
# load doc
raw_documents_lx = TextLoader('../data/dataset/wikipedia-en-sentences.txt', encoding='utf-8').load()

# split
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size =  1000,
    chunk_overlap = 100,
)
splits  = text_splitter.split_documents(raw_documents_lx)

# create vectorDB
embedding_model_dict = {
        "bge-large-zh": "../../../czh/text2mapper-exp/models/bge-large-zh"
    }
EMBEDDING_MODEL = "bge-large-zh"
embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL])

vectorstore = Chroma.from_documents(documents=splits, 
                                    embedding=embeddings,
                                    persist_directory="../data/vectorstore/wikipedia-en-sentences",
                                    )

retriever = vectorstore.as_retriever()

Description

I'm getting the above error when using the langchain for RecursiveCharacterTextSplitter，and my document is huge, it's a corpus of Wikipedia texts with 933 million English characters. My chunk_size = 1000 and chunk_overlap = 100,langchain=0.3.3.

and my error is this picture

System Info

langchain=0.3.3