AI21: Jamba Mini 1.7

AI21's Jamba Mini 1.7, a compact model within the Jamba family, demonstrates strong performance in terms of speed and cost-effectiveness. It typically performs among the fastest models, ranking in the 75th percentile for speed, and offers competitive pricing, ranking in the 71st percentile. The model boasts a significant 256K context window, a key feature of its SSM-Transformer hybrid architecture. However, Jamba Mini 1.7 exhibits notable weaknesses across several accuracy benchmarks. Its performance in Hallucinations (24.0% accuracy), General Knowledge (16.7%), Ethics (7.1%), and Mathematics (5.0%) is consistently in the lower percentiles, indicating challenges in factual recall, ethical reasoning, and mathematical problem-solving. While it shows a relatively better, though still low, accuracy in Instruction Following (31.9%) and Reasoning (32.0%), these areas also suggest room for improvement. A particular strength appears to be Email Classification, where it achieves 90.0% accuracy, although this still places it in the 17th percentile compared to other models. The model's description highlights improvements in grounding and instruction-following, but the benchmark results suggest these are still developing areas.