EleutherAI: Llemma 7b

EleutherAI's Llemma 7B, a language model specialized in mathematics, consistently ranks among the fastest models and offers highly competitive pricing across five benchmarks. Created on April 14, 2025, and initialized with Code Llama 7B weights, it was further trained on the Proof-Pile-2 for 200B tokens, aiming for strength in mathematical reasoning and tool use. Despite its specialized training, Llemma 7B demonstrates significant weaknesses in general-purpose benchmarks. Its accuracy is notably low across all evaluated categories: General Knowledge (7.5%), Ethics (7.0%), Email Classification (8.0%), and Coding (4.0%). The model completely failed the Instruction Following benchmark with 0.0% accuracy. While its cost per evaluation is generally competitive, its duration for these tasks is often in the lowest percentiles, indicating slow processing relative to its low accuracy. This suggests that while the model is fast in general terms, its performance on these specific tasks is inefficient. The provided data does not include reliability ranking information, so no comment can be made on that aspect. Llemma 7B's key strength lies in its foundational training for mathematical reasoning and tool use, which is not directly reflected in these general benchmarks. Its notable weakness is its poor performance on broad-spectrum tasks, indicating a highly specialized, rather than general-purpose, capability.