Worth flagging the main constraint on late chunking: the whole document has to fit in the embedding model's context window (8K tokens for the jina-embeddings-v2 models the original paper used). Past that you have to split into macro-chunks first, so you're back to choosing boundaries, just at a coarser level.
The payoff is real for reference-heavy text. Under fixed-size splitting, a chunk like "its revenue grew 3%" loses which company "its" means, while late chunking keeps that because every token was embedded with the full document in view. Curious whether anyone has measured agentic chunking against plain recursive splitting plus a reranker, since a good reranker often recovers most of what clever boundaries buy you.
Agentic chunking is the one I'd want a record for, not just a better split. The agent reads the file, decides it is a contract or a table, and picks the method, but retrieval later only sees the chunks. A contract clause cut on paragraph length and a table cut on rows then look the same in the index, and a bad answer can't be traced to that choice. I'd store the label and the splitter next to each chunk, so the next run can tell a wrong method from a weak embedding.
The version is what makes that re-chunk selective. A label of contract plus paragraph splitter still looks the same after a bugfix to the splitter, so the index can't tell the old chunks from the new ones. I'd key the job on doc id, splitter id, and splitter version, and only rewrite the docs whose stored version is behind the one that just shipped.
Thanks for sharing. Keying the job on version means a bugfix only touches the docs that need it, which is a lot cheaper than re-chunking everything. Bonus, it gives you a clean rollback if the new splitter turns out worse.
Late chunking has a catch for some corpora. In the Jina paper it stopped helping on their needle and passkey tests, where the answer is a short fact buried in unrelated text, since the context it blends into each chunk is just noise there. It also only sees as far as the embedding window, 8,192 tokens for jina-embeddings-v2-small, so longer docs get cut into overlapping macro chunks first.
Great explanation 👏. I have one question in the Agent skills. Does the LLM itself execute the scripts, or does it generate the code and request the agent runtime or orchestrator to execute it using a Python interpreter or another tool?
One thing that's easy to miss: your chunk size should match your query type, not just your document type. Short factual lookups want small, dense chunks; multi-hop reasoning questions need bigger chunks with more surrounding context or the agent loses the thread between retrieval and generation. Worth testing chunk size against actual query logs, not just against the corpus.
One bad chunk can turn a useful AI agent into an expensive liability.
If a support agent retrieves half a policy or misses a key contract condition, it can give customers the wrong answer, trigger costly rework, or expose the business to compliance risk.
In production, retrieval errors don’t stay technical. They become business costs.
Agentic chunking hands the agent the question of what counts as one idea in a contract versus a table. That is an editorial decision, and in Designing Intelligence I am arguing it deserves a human owner, not a default.
Worth flagging the main constraint on late chunking: the whole document has to fit in the embedding model's context window (8K tokens for the jina-embeddings-v2 models the original paper used). Past that you have to split into macro-chunks first, so you're back to choosing boundaries, just at a coarser level.
The payoff is real for reference-heavy text. Under fixed-size splitting, a chunk like "its revenue grew 3%" loses which company "its" means, while late chunking keeps that because every token was embedded with the full document in view. Curious whether anyone has measured agentic chunking against plain recursive splitting plus a reranker, since a good reranker often recovers most of what clever boundaries buy you.
A reranker is probably easier to swap later too, since changing the chunking means re-embedding the whole corpus.
Agentic chunking is the one I'd want a record for, not just a better split. The agent reads the file, decides it is a contract or a table, and picks the method, but retrieval later only sees the chunks. A contract clause cut on paragraph length and a table cut on rows then look the same in the index, and a bad answer can't be traced to that choice. I'd store the label and the splitter next to each chunk, so the next run can tell a wrong method from a weak embedding.
Yep, and I'd keep the chunker version next to that label too. Then when you change the method you know exactly which docs need re-chunking.
The version is what makes that re-chunk selective. A label of contract plus paragraph splitter still looks the same after a bugfix to the splitter, so the index can't tell the old chunks from the new ones. I'd key the job on doc id, splitter id, and splitter version, and only rewrite the docs whose stored version is behind the one that just shipped.
Thanks for sharing. Keying the job on version means a bugfix only touches the docs that need it, which is a lot cheaper than re-chunking everything. Bonus, it gives you a clean rollback if the new splitter turns out worse.
Late chunking has a catch for some corpora. In the Jina paper it stopped helping on their needle and passkey tests, where the answer is a short fact buried in unrelated text, since the context it blends into each chunk is just noise there. It also only sees as far as the embedding window, 8,192 tokens for jina-embeddings-v2-small, so longer docs get cut into overlapping macro chunks first.
Makes sense. Blend a short fact with a page of unrelated text and you're basically averaging it away.
Great explanation 👏. I have one question in the Agent skills. Does the LLM itself execute the scripts, or does it generate the code and request the agent runtime or orchestrator to execute it using a Python interpreter or another tool?
As far as I know the model only writes the command or code, and the agent runtime is what actually runs it and hands the output back.
Hi,
I started out on Substack. Just crossed 200 subs, hoping for the best. Latest post: https://engineerx0.substack.com/p/model-context-protocol-explained
Complex chunking looks sharp in diagrams, but real production dies on latency, not how you cut text.
Fair, though most of the fancy chunking cost lands at index time. The query path only slows down if you bolt on extra steps like reranking.
Changing the API key is easy, but re-testing edge cases and fixing broken outputs in production is where the real headache starts.
And the edge cases are always the ones nobody wrote down. Keeping a running list of weird failures beats retesting from memory.
One thing that's easy to miss: your chunk size should match your query type, not just your document type. Short factual lookups want small, dense chunks; multi-hop reasoning questions need bigger chunks with more surrounding context or the agent loses the thread between retrieval and generation. Worth testing chunk size against actual query logs, not just against the corpus.
chunking sounds simple until retrieval starts failing in weird ways. which of the eight strategies is your default for mixed long-form documents?
One bad chunk can turn a useful AI agent into an expensive liability.
If a support agent retrieves half a policy or misses a key contract condition, it can give customers the wrong answer, trigger costly rework, or expose the business to compliance risk.
In production, retrieval errors don’t stay technical. They become business costs.
agentic chunking picking a different split for contracts vs code is clever. does the agent ever get it badly wrong?
Agentic chunking hands the agent the question of what counts as one idea in a contract versus a table. That is an editorial decision, and in Designing Intelligence I am arguing it deserves a human owner, not a default.