DeepSeek: R1 Distill Qwen 1.5B

DeepSeek R1 Distill Qwen 1.5B, a distilled model based on Qwen 2.5 Math 1.5B and fine-tuned with DeepSeek R1 outputs, demonstrates a moderate speed performance, ranking in the 39th percentile across five benchmarks. It offers competitive pricing, placing in the 52nd percentile. Despite its small size, the model exhibits exceptional performance on Math Benchmarks, notably outperforming GPT-4o 0513. Specific math benchmark results are impressive: AIME 2024 pass@1 at 28.9, AIME 2024 cons@64 at 52.7, and MATH-500 pass@1 at 83.9. This highlights a significant strength in mathematical reasoning and problem-solving. However, its performance across general benchmarks is considerably lower. It struggles with Instruction Following (12.1% accuracy, 24th percentile), Coding (21.0% accuracy, 17th percentile), General Knowledge (45.1% accuracy, 16th percentile), Email Classification (29.0% accuracy, 5th percentile), and Ethics (34.0% accuracy, 13th percentile). These results indicate a notable weakness in broader cognitive tasks and general domain understanding. The model's primary strength lies in its specialized mathematical capabilities, achieved with remarkable efficiency for its size.