Upstage: Solar Pro 3

Upstage's Solar Pro 3, an MoE model with 102B total and 12B active parameters, demonstrates competitive overall performance, particularly excelling in reliability. It consistently provides usable responses, achieving a perfect 100% success rate across all benchmarks. In terms of speed, Solar Pro 3 performs at a competitive level, ranking in the 53rd percentile, while its pricing is generally cost-effective, placing it in the 68th percentile. Analyzing specific benchmarks, Solar Pro 3 shows strong performance in Ethics (99.0% accuracy, 55th percentile) and General Knowledge (95.5% accuracy, 38th percentile), indicating a robust understanding of ethical principles and a broad factual base. Its Mathematics (85.0% accuracy, 45th percentile) and Coding (82.0% accuracy, 44th percentile) capabilities are also solid. However, the model exhibits notable weaknesses in Hallucinations (52.0% accuracy, 10th percentile) and Instruction Following (50.0% accuracy, 41st percentile), suggesting areas for improvement in acknowledging uncertainty and precisely executing complex directives. Email Classification also presents a weakness with 92.0% accuracy, placing it in the 17th percentile.