NousResearch: Hermes 2 Pro - Llama-3 8B

NousResearch: Hermes 2 Pro - Llama-3 8B, released on May 26, 2024, demonstrates strong performance in several key areas. The model consistently ranks among the fastest, achieving the 86th percentile across eight benchmarks, and offers highly competitive pricing, placing in the 93rd percentile. Its reliability is exceptional, boasting a 97% success rate, indicating minimal technical failures. In terms of specific benchmarks, Hermes 2 Pro excels in acknowledging uncertainty, achieving 96.0% accuracy in Hallucinations (Baseline) and being noted as the most accurate model at its price point for this category. It also performs well in Ethics (Baseline) with 98.0% accuracy and Email Classification (Baseline) at 95.0% accuracy, both with very competitive costs. However, the model shows notable weaknesses in complex reasoning tasks, scoring 24.0% in Reasoning (Baseline), and in advanced Mathematics (Baseline) with 48.0% accuracy. Instruction Following (Baseline) and Coding (Baseline) also present areas for improvement, with accuracies of 38.4% and 51.0% respectively. Its General Knowledge (Baseline) is moderate at 91.8%. Overall, its strengths lie in cost-efficiency, speed, reliability, and specific classification and ethical understanding tasks, while complex problem-solving and advanced academic domains represent areas for further development.