Mistral: Devstral Small 1.1

Devstral Small 1.1, a 24B parameter open-weight language model for software engineering agents, demonstrates a balanced performance profile with notable strengths in specific areas. It performs among the fastest models, ranking in the 78th percentile for speed across various benchmarks, and consistently offers highly competitive pricing, placing in the 83rd percentile. The model exhibits exceptional reliability, achieving a 100% success rate across all evaluated benchmarks, indicating robust operational stability. In terms of benchmark performance, Devstral Small 1.1 shows strong capabilities in Hallucinations (92.0% accuracy) and General Knowledge (97.5% accuracy), though these scores place it around the 45-48th percentile. Its Ethics performance is also solid at 98.0% accuracy. A key strength lies in its Coding capabilities, achieving 85.0% accuracy and ranking in the 52nd percentile, further supported by its impressive 53.6% on SWE-Bench Verified, surpassing other open models. Email Classification is another strong suit at 98.0% accuracy (68th percentile). Weaknesses are observed in Instruction Following (51.0% accuracy) and Reasoning (64.0% accuracy), where it ranks in the lower 40th percentiles. Despite these areas for improvement, its overall efficiency and reliability make it a compelling choice for agentic coding workflows.