Google: Gemma 3 12B

Google's Gemma 3 12B, a multimodal AI model supporting vision-language input and text outputs, demonstrates strong overall performance with notable strengths in speed, pricing, and reliability. It consistently ranks among the fastest models, achieving an Infinityth percentile across 8 benchmarks, and offers highly competitive pricing, placing in the 87th percentile across 7 benchmarks. The model exhibits exceptional reliability with a 97% success rate, indicating minimal technical failures. In terms of specific benchmarks, Gemma 3 12B excels in Email Classification, achieving 99.0% accuracy and being the most accurate among models of its speed. It also performs well in General Knowledge (97.0% accuracy) and Ethics (98.0% accuracy), though these fall within the mid-range percentile for accuracy. Its Coding capabilities are solid at 85.0% accuracy. However, the model shows significant weaknesses in Instruction Following, with one benchmark yielding 0.0% accuracy and another at 19.0%. Mathematics and Reasoning also present areas for improvement, with accuracies of 67.0% and 59.2% respectively, placing them in lower percentiles. The model's ability to handle a 128k token context window and support over 140 languages are key features, alongside improved math, reasoning, and chat capabilities, including structured outputs and function calling.