NVIDIA: Nemotron 3 Ultra

NVIDIA Nemotron 3 Ultra demonstrates moderate speed performance, ranking in the 31st percentile across benchmarks, and offers competitive pricing, placing it in the 50th percentile. The model excels in specific areas, achieving perfect accuracy in both Instruction Following and Email Classification. For these tasks, it stands out as the most accurate model at its price point and among models of comparable speed. However, its performance varies significantly across other categories. It shows a reasonable ability to acknowledge uncertainty, with 95.2% accuracy in the Hallucinations benchmark. Conversely, the model exhibits notable weaknesses in complex reasoning tasks, scoring only 39.3% in Reasoning, and particularly struggles with specialized knowledge and problem-solving. Its accuracy in Coding (22.9%), General Knowledge (28.6%), Ethics (7.7%), and Mathematics (15.0%) is considerably low, placing it in the lower percentiles for these categories. The model's architecture, a hybrid Transformer-Mamba mixture-of-experts with 55B active parameters, suggests a design optimized for certain types of tasks, while indicating areas for further development in others.