mongodb: [performance] Increase DEFAULT_INSERT_BATCH_SIZE to 100,000 and introduce sizing constraints

[Jibola]

Thank you for contributing to LangChain!

Increase DEFAULT_INSERT_BATCH_SIZE to 100,000 and introduce sizing constraints

• [x]

  • Description: Increasing our default batch size to improve performance
  • Issue: N/A
  • Dependencies: None
  • Twitter handle: @mongodb

• Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

• Make sure optional dependencies are imported within a function.
• Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
• Most PRs should not touch more than one package.
• Changes should be backwards compatible.
• If you are adding something to community, do not re-import it in langchain.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		May 14, 2024 10:10pm

[ShaneHarvey]

The one concern here is what happens when someone calls add_texts() with a generator that yields a large amount of data. With this new code we'll inflate the entire thing into memory which can degrade performance. For example:

add_texts((random_large_string() for i in range(1_000_000))

In this case it would be best to limit the batches roughly based on the string size in addition to batch_size. We could do that by tracking the sum total of the batch's text:

                size += len(text)
                texts_batch.append(text)
                metadatas_batch.append(metadata)
                if (i + 1) % batch_size == 0 or size >= 47_000_000:

47MB is the maxMessageSizeBytes that pymongo can batch in one message minus 1MB of overhead to account for other bson data.

[Jibola]

Yeah, that's fine with me. We'll re-introduce BATCH_SIZE as 100_000 and also have an additional bounding of 47_000_000 on the text aggregate text length. In this case, then, should we also include the length of the accompanying metadata?

[ShaneHarvey]

LGTM!