To understand what techniques are adopted in frontier labs to make AI applications more efficient, we met with the OpenAI engineers who developed and shipped various efficiency techniques into the systems behind Codex and ChatGPT Work.
Really fun and accessible article. The abstract goal of minimizing data processing without losing accuracy always sounds easy to the naive. This post is a good example of how much engineering, how much insight, and how much creativity is required to find and address inefficiencies. For example, the idea of greedy inference with a smaller model to batch up a proposal for grading by a large model requires a lot of work just to make the system not fail. Same for cache-aware routing.
Really fun and accessible article. The abstract goal of minimizing data processing without losing accuracy always sounds easy to the naive. This post is a good example of how much engineering, how much insight, and how much creativity is required to find and address inefficiencies. For example, the idea of greedy inference with a smaller model to batch up a proposal for grading by a large model requires a lot of work just to make the system not fail. Same for cache-aware routing.
The real AI skill isn't getting better answers. It's learning which answers deserve a second thought.