How SWE-Serve Exposes the Gap Between Local Tests and Live Serving
NVIDIA and SGLang developers introduced SWE-Serve, a benchmark containing 53 tasks that assess AI coding agents on inference-serving software. The evaluation targets failures where patches pass local unit tests but break live model serving. This highlights a gap in current agent capabilities, providing a new metric for evaluating reliability in production-like environments.
SWE‑Serve, a benchmark created by NVIDIA and the SGLang team, consists of 53 tasks that probe AI coding agents on the full inference‑serving stack. The suite reveals that patches which succeed on local unit tests can still break when the server actually loads a model and processes real requests. This mismatch points to a reliability shortfall in current agents when they are moved from isolated testing to production‑like environments. The measurement process runs each task through the complete serving pipeline, from model loading to handling a request via the public interface, and checks whether the returned output is correct. By comparing these end‑to‑end results with the outcomes of conventional unit tests, the benchmark quantifies the frequency of silent failures that only appear under live serving conditions. The benchmark does not capture performance metrics such as latency or resource consumption, nor does it evaluate how well an agent adapts to evolving model versions. It also cannot indicate which specific coding patterns are responsible for the observed gaps, leaving those details uncertain.