Dolphin3.0 Mistral 24B

Dolphin3.0 Mistral 24B, created by cognitivecomputations, demonstrates competitive response times, ranking in the 52nd percentile for speed across various benchmarks. A significant advantage is its consistently competitive pricing, placing it in the 89th percentile. The model also exhibits exceptional reliability, boasting a 97% success rate, indicating minimal technical failures and consistent evaluable responses. In terms of performance across categories, Dolphin3.0 shows a notable strength in Instruction Following, achieving perfect 100% accuracy in one benchmark, making it the most accurate among models of comparable speed. It also performs well in General Knowledge and Ethics, both scoring 97% accuracy. However, the model exhibits weaknesses in other areas. Its hallucination rate is relatively high at 86% accuracy, suggesting it struggles to appropriately acknowledge uncertainty. Furthermore, its performance in Reasoning (48% accuracy), Coding (63% accuracy), and Email Classification (93% accuracy) is in the lower percentiles, indicating areas for improvement. Another Instruction Following benchmark also showed a lower accuracy of 40.4%. Overall, Dolphin3.0 is a reliable and cost-effective option with strong instruction-following capabilities, but its performance in complex reasoning, coding, and hallucination mitigation could be enhanced.