Nous: DeepHermes 3 Mistral 24B Preview

Nous: DeepHermes 3 Mistral 24B Preview demonstrates a strong overall performance profile, particularly excelling in reliability and cost-effectiveness. It performs among the fastest models, ranking in the 67th percentile for speed, and offers competitive pricing, placing in the 73rd percentile. Notably, the model exhibits exceptional reliability with a perfect 100% success rate across all benchmarks, indicating consistent and stable operation. In terms of specific capabilities, DeepHermes 3 shows remarkable strength in Ethics, achieving 100% accuracy, making it the most accurate model at its price point and among models of similar speed. It also performs very well in General Knowledge (98.0% accuracy) and Email Classification (97.0% accuracy). Its Coding performance is solid at 83.0% accuracy. However, the model's Instruction Following (50.5% accuracy) and Reasoning (60.0% accuracy) capabilities are moderate, suggesting areas for potential improvement. Its hallucination rate is relatively low at 92.0% accuracy, indicating a reasonable ability to acknowledge uncertainty. The unique "deep thinking" mode, activated via a system prompt, offers a promising avenue for enhancing its reasoning capabilities, though its impact on these specific baseline scores is not directly quantifiable here.