Mistral: Pixtral 12B

Mistral: Pixtral 12B demonstrates strong performance in terms of operational efficiency, performing among the fastest models with a 79th percentile speed ranking and consistently offering competitive pricing, ranking in the 84th percentile. As the first multi-modal, text+image-to-text model from Mistral AI, its benchmark results reveal a mixed performance profile. The model shows a notable strength in Coding, achieving 70.0% accuracy, placing it in the 32nd percentile, and performing well on cost and duration for this category. However, its performance across other critical areas is significantly lower. It struggles considerably with Mathematics (1.0% accuracy), Email Classification (26.0% accuracy), Instruction Following (29.0% accuracy), and Reasoning (18.0% accuracy). Its ability to acknowledge uncertainty, as measured by the Hallucinations benchmark, is moderate at 66.0% accuracy, indicating some propensity to generate incorrect information rather than deferring. General Knowledge and Ethics also show room for improvement, with accuracies of 66.6% and 59.5% respectively. Overall, while Pixtral 12B excels in cost-effectiveness and speed, its accuracy across many cognitive and classification tasks indicates significant areas for development.