Mistral: Magistral Small 2506

Mistral: Magistral Small 2506, a 24B parameter instruction-tuned model, demonstrates competitive response times, performing among the faster models with a speed ranking in the 48th percentile. It also offers competitive pricing, ranking in the 42nd percentile. A notable strength is its exceptional reliability, ranking in the 92nd percentile, indicating consistent and usable responses with minimal technical issues. In terms of specific benchmarks, the model shows strong performance in Coding (85.0% accuracy, 68th percentile) and Email Classification (96.0% accuracy), though the latter comes with a higher cost and slower duration. Its Instruction Following capabilities are a standout, achieving a top 3 ranking in speed, despite a moderate 46.0% accuracy. However, the model exhibits weaknesses in Ethics (88.0% accuracy, 24th percentile), Reasoning (50.0% accuracy, 36th percentile), and General Knowledge (71.0% accuracy, 23rd percentile), where its accuracy and percentile rankings are lower. While generally competitive in speed and price, its performance varies significantly across different cognitive tasks, excelling in structured tasks like coding and classification but showing room for improvement in more abstract reasoning and broad knowledge domains.