Mistral: Mistral Small 3.1 24B

Mistral Small 3.1 24B Instruct, an upgraded 24-billion parameter model from mistralai, demonstrates a strong overall performance profile, particularly excelling in cost-efficiency and reliability. It performs among the fastest models, ranking in the top tier for speed (68th percentile), and consistently offers highly competitive pricing (88th percentile). Notably, the model exhibits exceptional reliability with a perfect 100% success rate across all benchmarks, indicating robust and consistent operation. In terms of specific benchmarks, Mistral Small 3.1 shows strong performance in acknowledging uncertainty, achieving 98.0% accuracy in Hallucinations (Baseline) tests. It also performs well in General Knowledge (98.7% accuracy) and Ethics (99.0% accuracy). A key strength is its Mathematics performance, where it achieves 86.0% accuracy, making it the most accurate model at its price point. However, the model shows moderate performance in more complex tasks such as Instruction Following (54.3% accuracy), Reasoning (64.0% accuracy), and Coding (81.0% accuracy), suggesting areas for potential improvement in these domains. Its Email Classification accuracy (96.0%) is also in the lower percentile compared to other models.