In this article, we are going to understand how this connection works and why a model needs help understanding the order of the words even when the words already arrive in an ordered list.
Fourier's cosines also sit inside every JPEG, deciding which detail of a photograph gets thrown away. The same math now tells a model where a word sits. My photography work compares that line, from deterministic digital images to probabilistic ones.
Since I mostly work with the money side of software careers, I notice the same lesson plays out in retirement planning: a number carrying information isn't the same as a number the system actually uses.
Most engineers track their corpus, but never account for the sequence in which goals arrive. A house at 38, a child's education at 45, retirement at 60 — same savings, completely different answers.
A model will take a 128k input without one error and still miss facts buried in it. RULER tested 17 models that all claim 32k or more, and only half held up at 32k, even though nearly all of them ace plain needle-in-a-haystack. I'd run a harder retrieval test at the length you actually serve before raising max context.
The relative-shift property is the part that clicks for me: advancing two positions always rotates a given frequency pair by the same angle, whether you start at 0 or 200. That reusable offset is exactly what absolute integer positions don't give you for free. One practical follow-on: when people swap sinusoidal PE for learned embeddings on short contexts, they often quietly lose that translation structure — curious whether you've seen RoPE's relative bias matter more for code (bracket matching across long spans) than for prose in the systems you covered.
The Alice-trusts-Bob example is a great way to show why order matters. This same problem shows up in RAG pipelines where chunk ordering gets lost during retrieval and the model treats the reassembled context like a bag of facts instead of a sequence.
Fourier's cosines also sit inside every JPEG, deciding which detail of a photograph gets thrown away. The same math now tells a model where a word sits. My photography work compares that line, from deterministic digital images to probabilistic ones.
Since I mostly work with the money side of software careers, I notice the same lesson plays out in retirement planning: a number carrying information isn't the same as a number the system actually uses.
Most engineers track their corpus, but never account for the sequence in which goals arrive. A house at 38, a child's education at 45, retirement at 60 — same savings, completely different answers.
Order changes the outcome in both cases.
A model will take a 128k input without one error and still miss facts buried in it. RULER tested 17 models that all claim 32k or more, and only half held up at 32k, even though nearly all of them ace plain needle-in-a-haystack. I'd run a harder retrieval test at the length you actually serve before raising max context.
The relative-shift property is the part that clicks for me: advancing two positions always rotates a given frequency pair by the same angle, whether you start at 0 or 200. That reusable offset is exactly what absolute integer positions don't give you for free. One practical follow-on: when people swap sinusoidal PE for learned embeddings on short contexts, they often quietly lose that translation structure — curious whether you've seen RoPE's relative bias matter more for code (bracket matching across long spans) than for prose in the systems you covered.
The Alice-trusts-Bob example is a great way to show why order matters. This same problem shows up in RAG pipelines where chunk ordering gets lost during retrieval and the model treats the reassembled context like a bag of facts instead of a sequence.