AI21: Jamba Mini 1.6

AI21 Jamba Mini 1.6 demonstrates strong performance in terms of efficiency, consistently ranking among the fastest models with a 73rd percentile speed ranking across five benchmarks. It also offers competitive pricing, placing in the 70th percentile for cost-effectiveness. The model's hybrid architecture, combining State Space Models and Transformer attention, is designed for extremely long-context tasks, supporting up to 256K tokens, and boasts superior inference efficiency. However, its accuracy across various benchmarks is a notable weakness. In General Knowledge, it achieved only 2.0% accuracy (10th percentile), and in Ethics, 34.0% (13th percentile). Coding performance was also low at 7.0% accuracy (14th percentile). While Email Classification showed a respectable 90.0% accuracy, this still placed it in the 17th percentile. Instruction Following was moderate at 41.0% accuracy (40th percentile). Despite these accuracy challenges, its efficiency in terms of duration and cost for these tasks remains competitive. Key strengths lie in its architectural innovation for long-context processing, multilingual support, structured JSON output, and tool-use capabilities, making it potentially valuable for specific applications where these features are paramount, even with current accuracy limitations.