From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding

JetBrains engineers evaluate coding agents beyond simple pass rates by analyzing execution trajectories. Their study compares Claude Opus 4.7 and Gemini 3.5 Flash, revealing identical task resolution but divergent efficiency. Opus used fewer steps at higher cost, while Gemini required more steps for lower cost. This highlights the need for multi-dimensional agent evaluation metrics.