| Livenerf: Has Opus 5.5 been nerfed yet?(github.com) | |
| 911 points by bryan0 14 days ago | 388 comments | |
tl;dr: Livenerf is an append-only benchmark that tests whether Claude Opus 5.5 degrades after launch, using a frozen panel of 78 "sometimes right" questions from GPQA Diamond, MMLU-Pro, and competition math run daily through a pinned Claude Code CLI. It uses pre-registered statistics (paired deltas with clustered SEs), a control arm running Opus 5, and a decision rule requiring a 3+ point change clearing 99% intervals in two consecutive 10-day windows. Validation shows it can detect lower reasoning effort but can't distinguish a same-family model swap in one window's samples. | |
HN Discussion:
| |