Anthropic: Claude 3 Haiku

Anthropic's Claude 3 Haiku, launched on March 12, 2024, is positioned as a fast and compact model with a substantial 200,000 context length. It performs among the fastest models, typically ranking in the top tier for speed (80th percentile), and offers competitive pricing (58th percentile). The model demonstrates exceptional reliability with a 97% success rate, indicating consistent and usable responses. In terms of performance across benchmarks, Claude 3 Haiku shows strong capabilities in Email Classification (98.0% accuracy, 65th percentile) and Instruction Following (61.0% accuracy, 66th percentile), suggesting proficiency in structured tasks and adherence to directives. Its General Knowledge (96.5% accuracy) and Ethics (98.0% accuracy) scores are also high, though their percentile rankings are moderate. A notable weakness is its performance in Reasoning (36.0% accuracy, 24th percentile) and Mathematics (58.0% accuracy, 26th percentile), indicating challenges with complex problem-solving and advanced mathematical concepts. Hallucination accuracy is moderate at 86.0%, placing it in the lower quartile for this metric. Coding performance is also in the lower half (72.0% accuracy, 34th percentile). Overall, Claude 3 Haiku excels in speed, reliability, and specific classification/instruction-based tasks, while showing areas for improvement in complex reasoning and mathematical domains.