Is it possible to implement Late chunking in LangChain text splitters?

[leo-gan]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

It is an interesting new approach for chunking.

Motivation

The semantic chunking should be improved.

Proposal (If applicable)

No response

[phuonganhcorn]

I'm very interested in this topic since I agree that Semantic Chunking should be improved. During my research, I found that the logic behind the Late Chunking technique might solve many of the performance issues caused by traditional chunking methods.

I have outlined a flow to manually implement the Late Chunking technique in Langchain using the features currently supported by Langchain (I haven't tested it yet, but I plan to do so soon and share the results later).

---

1. I will use Langchain's "Split by tokens" feature to split the document into tokens.
2. Since the chunk size is manually set, I will set it quite small (10-20 tokens per chunk, as this balance provides enough context without being too verbose).
3. Then, I will embed these small "chunks."
4. I will retrieve these "chunks" and group them to create the closest match to my query. This step involves finding chunks with the highest semantic + keywords relevance to the query.
5. After that, since using discrete tokens might create nonsense or unclear information, I will use an LLM to rewrite the content that I retrieved.

---

The flow I outlined above is just a concept I'm exploring right now. The main challenge is determining the optimal chunk size, so I'll be testing this soon and will share the results once I have them.

I’d greatly appreciate any recommendations for improving my approach or suggestions for alternative methods that might work better!

[ognjenAct]

Hey guys, same here, it would be nice to see implementation for late chunking, we tried our custom flow similar to what is already described here but it was really weak. Anyone has idea?

Tnx

[a1ix2]

I'm in the same boat here. I'm especially interested in late-chunking in the context of addressing the "haystack in a haystack" problem that plagues scholarly RAGs.

jina.ai gives an example notebook here and makes the code available here, but it's quite basic and custom and not readily compatible with typical RAG workflows using vector databases and so on.

[seroetr]

is there any development with regard to topic? It's been a long time since the opening time of discussion. It will be vital for late chunking to be integrated into langchain system. Could you give us some information?

[leo-gan]

See #27278

[seroetr]

Hi Leo-gan,

Thansk for directing me to related part which is
https://langchain-git-fork-leo-gan-long-context-embedding-langchain.vercel.app/docs/how_to/long_context_embedding/

Does this approach really mirror JinaAI's late chunking method?

There are two main issues in langchain approach:

1. Context Usage: JinaAI's late chunking preserves full-context embeddings by embedding the entire sequence and chunking later, retaining context across chunks. LangChain's method instead generates embeddings for each word incrementally based on its preceding context, which is a form of progressive context-aware chunking, but not true late chunking.

2. Difference in Pooling: In JinaAI, embeddings are computed in chunks, and pooling is done after the embedding process. In LangChain's implementation, it's pooling individual words' embeddings progressively.

Will there be any official langchain integration of late chunking?

[a1ix2]

There's a little bit of discrepancy between how JinaAI presents late chunking in the blog and how the algorithm they suggest actually operate for example in the original paper. The original paper does chunking after embedding each token despite the blog post hyping the use of JinaAI's embedding model having a 8192 context size. In other words context-awareness is entirely built out of pooling operations on token embeddings. The second embedding method they present, called long late chunking, use progressive pooling of more and more token embeddings until some chunk size is reached, or whatever else context delimiter of one's choice is reached.

With embedding models with large context sizes, there's nothing preventing late-chunking to work on embeddings at the sentence level, or paragraph level, or whatever and do pooling on those instead, which is what the blog post seems to suggest.

I guess what I'm getting at is that (long) late chunking by itself is more of a concept on how to accelerate building context-aware embeddings using progressive pooling, and not a specific algorithm restricted to chunking of token level embeddings.

In my mind, working on academically oriented RAG using scholarly material/PDFs, taking advantage of embedding models with large context sizes suggests a late chunking approach where chunking happens at the sentence or paragraph level, and then do spit out embeddings while progressive pooling using chunks of those over full sections up to the full article depending on the context-size of the downstream LLM. Whether this works better than working with token level embeddings I have no idea.

[leo-gan]

As I mentioned in the PR, the Long Context Embedding term is more suitable than Late Chunking term.
The blog is about the method of applying embedding, not about how to chunk the text. The Late Chunking is misleading, IMHO.

Enriching the chunk embedding with the surrounding context can be done using different methods.

[Aillian]

Have anyone implemented the code?

[leo-gan]

See #27278 #27278
Disclamer: it is not an update in LangChain but the implementation example and discussions