The tokenizer finding is a pricing problem nobody in this industry has had to think about before.
If the same document costs up to fifteen times more tokens depending on the language it's written in, then your cost to serve varies by market for reasons that have nothing to do with what the customer is doing. A German or Turkish user costs more than an English one for identical work, and gets less usable context for the same money.
Every SaaS company prices per seat and eats the difference in cost to serve. That's fine when marginal cost is near zero. Here it isn't, so you either price by language, which is commercially impossible, or you subsidise some markets from others and accept that your worst-margin customers are also your least well-served ones. Nobody's publishing that per-market gross margin table, and I'd love to see it.
The caching rule has the same shape. Stable content at the top, changing content at the bottom, cached prefix at a tenth of the rate. That's an architectural decision with a direct line to gross margin, which means prompt structure is now a finance concern. Not a sentence I expected to write.
we experience a "conversation" but the model is really re-reading the whole transcript cold on every single turn. Once that clicks, the growing cost per turn, the dropped details, and even "why did editing an old message change the answer?" all stop being mysterious.
The tokenizer finding is a pricing problem nobody in this industry has had to think about before.
If the same document costs up to fifteen times more tokens depending on the language it's written in, then your cost to serve varies by market for reasons that have nothing to do with what the customer is doing. A German or Turkish user costs more than an English one for identical work, and gets less usable context for the same money.
Every SaaS company prices per seat and eats the difference in cost to serve. That's fine when marginal cost is near zero. Here it isn't, so you either price by language, which is commercially impossible, or you subsidise some markets from others and accept that your worst-margin customers are also your least well-served ones. Nobody's publishing that per-market gross margin table, and I'd love to see it.
The caching rule has the same shape. Stable content at the top, changing content at the bottom, cached prefix at a tenth of the rate. That's an architectural decision with a direct line to gross margin, which means prompt structure is now a finance concern. Not a sentence I expected to write.
Latency insides a chatbot are an ops tell. The investment question is still which infrastructure bottlenecks get the durable rents.
Thank you! Very informative, and I love the diagrams.
Yep! This is why long chats get slow!
we experience a "conversation" but the model is really re-reading the whole transcript cold on every single turn. Once that clicks, the growing cost per turn, the dropped details, and even "why did editing an old message change the answer?" all stop being mysterious.