docs[experimental]: Make docs clearer and add min_chunk_size

[tibor-reiss]

Fixes #26171:

• added some clarification text for the keyword argument breakpoint_threshold_amount
• added min_chunk_size: together with breakpoint_threshold_amount, too small/big chunk sizes can be avoided

Note: the langchain-experimental was moved to a separate repo, so only the doc change stays here.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 1, 2024 6:11am

[efriis]

could you add some unit tests? I believe this has a bug where it will drop chunks if some of the final chunks are shorter than min_chunk_size

This is also relatively achievable as a postprocessing step, so not confident this needs to be part of this implementation in particular.

e.g.

def merge_short_docs(docs, min_chunk_size = 20):
    rtn = []
    doc = None
    for d in docs:
        if doc is None:
            doc = d
        else:
            # use first doc's metadata
            doc = Document(doc.page_content + d.page_content, metadata=doc.metadata)

        if len(doc.page_content) < min_chunk_size:
            continue
        rtn.append(doc)
        doc = None
    if doc is not None:
        rtn.append(doc)
    return rtn

note this allows you to configure metadata handling as well if you want to merge in a custom way.

Thoughts on documenting this instead of adding the param?

[tibor-reiss]

@efriis sure, I will try to add some tests. However, since start_index is not incremented, the second if will catch this:

# The last group, if any sentences remain
if start_index < len(sentences):

[tibor-reiss]

@efriis Test added. The current implementation does not miss the last sentence (or what is left) and does the same thing as your snippet just in less lines of code :)

However, I like the idea of adding a postprocessing step. On the other hand, it's not that simple to cover all edge cases - e.g. if the last chunk is smaller than min_chunk_size, it still remains as such. But if I would add it to the previous, then that previous chunk could grow too much, and so on. That was also the reason I did not add a max_chunk_size, because this can be already controlled with breakpoint_threshold_amount.
Let me know what you think.

[tibor-reiss]

@efriis is there a way to restart vercel, please? Or how can I access the logs/build from this workflow? I am getting 404.

[efriis]

hey sorry for the delay!

feel free to reopen the code changes against the langchain-experimental repo (this package moved)! https://github.com/langchain-ai/langchain-experimental

when you merge in master, the vercel build should work better. (some issues with the vercel build last week that have been fixed by upgrading to docusaurus 3)

[tibor-reiss]

Hi @efriis, as requested, I moved the code change to the other repo, leaving just the doc change here.

[tibor-reiss]

Friendly ping @baskaryan @eyurtsev
The corresponding has been live in experimental for some time now