In this article, we’ll look at how LLMs actually process the information you give them, what context engineering is, and the strategies that can help with it.
“The quality of AI’s output tells you something about the quality of your input.
Think of it like a detective solving a case. One clue (the suspect was in town that day) leaves hundreds of possibilities open. Two clues (they were in town and had a motive) narrows it down. Five independent clues might point to exactly one person.” - Julie Zhuo
The "lost in the middle" problem is something I've hit repeatedly. My agent setup has an instruction file that grew to about 4,000 tokens over time - added sections as needed - and at some point noticed it was ignoring rules buried in the middle.
Ended up restructuring: most critical rules at top, less frequent edge cases at bottom. Helped immediately. The point about writing information externally to overcome statelessness is also the part I'd flag for anyone starting out - platform memory doesn't travel across models or cron runs. A flat markdown file does. Took me longer than it should've to figure that out.
The framing of context engineering as four strategies (write, select, compress, isolate) is really useful.
I think the "isolate" pattern is the one most teams underestimate. It's counterintuitive that splitting work across multiple smaller contexts outperforms one large context using a more powerful model, but Anthropic's 90% improvement with multi-agent delegation makes the case pretty clearly.
The accuracy cliff at long context is the most underreported finding of the year. Everyone is racing to expand context windows while the models quietly get worse the more you give them. The implication for retrieval architectures is the opposite of what most teams are building. Less context, better selected, beats more context every time.
There’s something almost poetic about a system that becomes less precise as it’s given more to think about. You can feel the attention thinning out, like a crowd that grows so large it forgets why it gathered in the first place.
Most people assume intelligence compounds with input, but here it behaves more like pressure. Add enough of it and the structure doesn’t strengthen, it distorts. At some point the signal isn’t lost, it’s buried under too many things pretending to be it.
The steep accuracy dropoff past context thresholds highlights why externalizing state is a critical architecture choice. In scaled agent workflows, robust eval loops prove that selective context hydration and schema-constrained working memory outperform simply dumping larger token windows into the model every time.
“The quality of AI’s output tells you something about the quality of your input.
Think of it like a detective solving a case. One clue (the suspect was in town that day) leaves hundreds of possibilities open. Two clues (they were in town and had a motive) narrows it down. Five independent clues might point to exactly one person.” - Julie Zhuo
great read. awesome breakdown of the terminology as well. considering the audience here, i think it’s great we all align on the terms as we discuss.
https://regulatingai.substack.com/p/ai-governance-now-your-weekly-compass-970?r=3pjruc&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
The "lost in the middle" problem is something I've hit repeatedly. My agent setup has an instruction file that grew to about 4,000 tokens over time - added sections as needed - and at some point noticed it was ignoring rules buried in the middle.
Ended up restructuring: most critical rules at top, less frequent edge cases at bottom. Helped immediately. The point about writing information externally to overcome statelessness is also the part I'd flag for anyone starting out - platform memory doesn't travel across models or cron runs. A flat markdown file does. Took me longer than it should've to figure that out.
The framing of context engineering as four strategies (write, select, compress, isolate) is really useful.
I think the "isolate" pattern is the one most teams underestimate. It's counterintuitive that splitting work across multiple smaller contexts outperforms one large context using a more powerful model, but Anthropic's 90% improvement with multi-agent delegation makes the case pretty clearly.
Great read, but I miss some mentions of context caching
Chroma’ research is pretty great, thanks for sharing!
The accuracy cliff at long context is the most underreported finding of the year. Everyone is racing to expand context windows while the models quietly get worse the more you give them. The implication for retrieval architectures is the opposite of what most teams are building. Less context, better selected, beats more context every time.
There’s something almost poetic about a system that becomes less precise as it’s given more to think about. You can feel the attention thinning out, like a crowd that grows so large it forgets why it gathered in the first place.
Most people assume intelligence compounds with input, but here it behaves more like pressure. Add enough of it and the structure doesn’t strengthen, it distorts. At some point the signal isn’t lost, it’s buried under too many things pretending to be it.
The steep accuracy dropoff past context thresholds highlights why externalizing state is a critical architecture choice. In scaled agent workflows, robust eval loops prove that selective context hydration and schema-constrained working memory outperform simply dumping larger token windows into the model every time.