Baidu: ERNIE 4.5 21B A3B Thinking

Baidu's ERNIE 4.5 21B A3B Thinking model demonstrates exceptional speed, consistently ranking among the fastest models across all benchmarks. It also offers highly competitive pricing, placing it among the most cost-effective options available. With an 87% success rate, the model exhibits strong reliability, indicating few technical issues during evaluation. Performance across benchmarks reveals a mixed but generally strong profile. The model excels in Reasoning, achieving 86.0% accuracy (75th percentile), aligning with its description of boosted reasoning depth. It also shows strong performance in General Knowledge (97.0% accuracy) and Ethics (98.0% accuracy). However, a significant weakness is observed in Instruction Following, where it scored 0.0% accuracy, suggesting a critical area for improvement. Coding performance is also low at 17.3% accuracy. Hallucinations are present at 80.0% accuracy, indicating room for improvement in acknowledging uncertainty. Email Classification and Mathematics show moderate performance at 96.0% and 68.0% accuracy, respectively. Overall, its key strengths lie in speed, cost-efficiency, and advanced reasoning capabilities, while instruction following and coding present notable weaknesses.