The judge scores whether claims are supported by the retrieved context stored in Hive, not by the wiki page as it stands when someone reads the report. A doc can be edited after that row was written, and the score still looks fine. I'd keep the document id and revision on the Hive row with the answer, so a later document-eval suggestion and the answer score refer to the same version rather than mixing a stale retrieval into the 48.9% helpfulness number.
We run a similar Slack bot at my company and it works really well because a smaller company only gets a handful of request types. What worries me is that even though I built it, the more it handles the fewer of those problems I ever work through myself. So when something it can't answer finally comes up I'm figuring out a part of the system I've never actually touched, which is pretty scary.
The split between the two failure modes is the most useful idea here: bad retrieval versus good retrieval with a bad answer. Since every chunk carries its source URL as its own sub-context, you could score them separately: did the right doc show up in the top-k at all, and does the cited passage actually support the claim. Did Uber's LLM-as-judge pipeline evaluate retrieval on its own, or only the final answer? A "Not Helpful" vote alone can't tell you which stage failed.
The citation link opens today's version of the page, while the answer came from whatever text the index held at the last ingest run. If the wiki changed in between, the engineer clicks through and finds a source that doesn't say what Genie said. Storing each page's last-edited time on its chunks and comparing it when the answer goes out catches that cheaply.
Checking answers before showing them to the user is the hard part in these text to SQL style systems. Did Uber use a separate validator step?
The judge scores whether claims are supported by the retrieved context stored in Hive, not by the wiki page as it stands when someone reads the report. A doc can be edited after that row was written, and the score still looks fine. I'd keep the document id and revision on the Hive row with the answer, so a later document-eval suggestion and the answer score refer to the same version rather than mixing a stale retrieval into the 48.9% helpfulness number.
We run a similar Slack bot at my company and it works really well because a smaller company only gets a handful of request types. What worries me is that even though I built it, the more it handles the fewer of those problems I ever work through myself. So when something it can't answer finally comes up I'm figuring out a part of the system I've never actually touched, which is pretty scary.
The split between the two failure modes is the most useful idea here: bad retrieval versus good retrieval with a bad answer. Since every chunk carries its source URL as its own sub-context, you could score them separately: did the right doc show up in the top-k at all, and does the cited passage actually support the claim. Did Uber's LLM-as-judge pipeline evaluate retrieval on its own, or only the final answer? A "Not Helpful" vote alone can't tell you which stage failed.
The citation link opens today's version of the page, while the answer came from whatever text the index held at the last ingest run. If the wiki changed in between, the engineer clicks through and finds a source that doesn't say what Genie said. Storing each page's last-edited time on its chunks and comparing it when the answer goes out catches that cheaply.