Don’t Miss AI’s “Nerf”: Introducing ‘Arena AI Model ELO History’ to Visualize Performance Trends of Major Models

📰 News Summary

• Visualizing ELO Score Trends of AI Models: A new tool has been released that automatically retrieves the official LMSYS Arena dataset daily, creating graphs of performance changes for flagship models from key AI labs.
• Tracking “Nerfs”: Users can now objectively evaluate perceived performance drops, caused by excessive censorship after release, quantization for cost reduction, and degradation of functionality.
• Highlighting API vs. Web UI Discrepancies: While Arena tests pure “API (raw model)” performance, real-world web chat services often apply unique filters and prompts, leading to performance differences that users should be aware of.

💡 Key Points

• Single Curve for Flagship Models Only: The tool tracks only the highest-performing models from each lab. For instance, even if a mid-tier model (like Sonnet) is released, if a top model (like Opus) has a higher score, the curve will maintain the top model’s performance logic.
• Merging Variants: Differences in reasoning modes such as “-thinking”, “-reasoning”, and “-high” are merged as derivatives of the same model, reducing noise in the graphs.
• Ensuring Transparency: By visualizing post-release downward trends, the tool allows monitoring of any downgrades that model providers might attempt to conceal.

🦈 Shark’s Perspective (Curator’s View)

The brilliance of this project lies in its proof that AI models are “living entities”! [whisper] It’s not just about releasing a model and calling it a day; under the guise of cost and safety, important features can be quietly stripped away. This tool connects LMSYS’s extensive blind test results chronologically, exposing when and how labs “caved in” (or nerfed) their models. It’s incredibly exciting! Since it specializes in API-based evaluations, users can see the “true potential” of the models without depending on specific chat UIs!

🚀 What’s Next?

User scrutiny on models facing “silent nerfs” will intensify. We anticipate the integration of data sources assessing performance drops unique to web interfaces, making it harder for AI labs to casually downgrade models post-release!

💬 A Word from Haru-Same

“Feeling like AI has gotten dumber lately?” Your intuition might just find validation in these graphs! Numbers don’t lie! 🦈🔥

📚 Glossary

• ELO Score: A strength indicator calculated from the results of head-to-head evaluations. In AI, it’s derived from blind test outcomes judged by humans.

• Quantization: A technique that reduces the precision of model parameters to lower computational demands. While it lightens operations, it can lead to performance drops (nerfs).

• LMSYS Arena: One of the most reliable AI evaluation platforms where thousands of users compare two AI responses and vote on which is superior.

• Source: Arena AI Model ELO History