Cohere: Command R (03-2024)

Cohere's Command-R (03-2024) model demonstrates strong performance in several key areas. It consistently ranks among the fastest models, placing in the 81st percentile across seven benchmarks, and offers competitive pricing, ranking in the 55th percentile. Notably, the model exhibits exceptional reliability with a 100% success rate across all benchmarks, indicating minimal technical failures and consistent provision of usable responses. In terms of specific benchmark performance, Command-R shows a strong aptitude for General Knowledge (92.5% accuracy) and Ethics (96.0% accuracy), suggesting robust understanding in these domains. Its ability to classify emails is also solid at 93.0% accuracy. However, the model struggles significantly with complex Reasoning (18.0% accuracy) and Instruction Following (33.0% accuracy), indicating areas for improvement in multi-step problem-solving and adherence to intricate directives. Its performance on Hallucinations (64.0% accuracy) suggests room for improvement in acknowledging uncertainty. While its Coding accuracy is moderate at 68.0%, its duration for this task is notably high, placing it in the 99th percentile for slowness.