Mistral: Devstral Small 2505

Mistral: Devstral Small 2505, a 24B parameter agentic LLM, demonstrates strong overall performance, particularly excelling in reliability and cost-efficiency. It boasts a perfect 100% success rate across all benchmarks, indicating exceptional reliability with no technical failures. In terms of speed, Devstral Small 2505 performs among the fastest models, ranking in the 73rd percentile. Its pricing is highly competitive, consistently placing in the 88th percentile for cost-effectiveness. Analyzing benchmark results, Devstral Small 2505 shows a standout performance in Instruction Following, achieving 100% accuracy in one instance, making it the most accurate among models of comparable speed. While its Coding (Baseline) accuracy is moderate at 82.0%, its cost and duration for this category are highly competitive. The model also exhibits strong performance in General Knowledge (98.0% accuracy) and Ethics (98.0% accuracy), though its percentile rankings in these areas are not as high. A notable weakness appears in another Instruction Following benchmark, where accuracy drops to 49.0%, suggesting some variability in complex instruction adherence. Reasoning and Email Classification show moderate accuracy. Overall, Devstral Small 2505 is a highly reliable and cost-effective model, particularly suited for agentic workflows and advanced software engineering tasks, with a strong aptitude for precise instruction following.