EleutherAI: Llemma 7b

EleutherAI's Llemma 7B, a mathematics-focused language model initialized from Code Llama 7B and trained on Proof-Pile-2, demonstrates exceptional speed and cost-efficiency. It consistently ranks among the fastest models and offers among the most competitive pricing across five benchmarks, achieving an "Infinityth percentile" in both categories. However, its performance on general benchmarks is notably low. In General Knowledge, it achieved only 7.5% accuracy (10th percentile), and in Ethics, 7.0% accuracy (8th percentile). Email Classification also showed limited capability with 8.0% accuracy (3rd percentile). A significant weakness is observed in Instruction Following, where it scored 0.0% accuracy, indicating a complete inability to follow complex directives. Coding performance was similarly low at 4.0% accuracy (10th percentile). These results suggest that while Llemma 7B excels in its core design for mathematical reasoning and tool use, its capabilities do not extend to broader general-purpose tasks. Its strengths are highly specialized, aligning with its description as a model for mathematics, particularly in chain-of-thought reasoning and computational tool integration.