StepFun: Step 3.7 Flash

Step 3.7 Flash, StepFun's 196B-parameter multimodal Mixture-of-Experts model, demonstrates exceptional performance across several key metrics. It consistently ranks among the fastest models available, achieving an Infinityth percentile in speed across eight benchmarks. Furthermore, its pricing is highly competitive, placing it in the 86th percentile across six benchmarks. The model exhibits outstanding reliability with a 98% success rate, indicating minimal technical failures. Analysis of benchmark results reveals significant strengths in several areas. Step 3.7 Flash achieved perfect accuracy in General Knowledge, Email Classification, Reasoning, Ethics, and Mathematics, often being the most accurate model at its price point and speed. Notably, it was the #1 model in accuracy for Mathematics, showcasing an excellent accuracy-to-cost ratio. Its Coding performance is solid at 85.7% accuracy. However, the model exhibits critical weaknesses in Hallucinations and Instruction Following, scoring 0.0% accuracy in both. This indicates a significant challenge in acknowledging uncertainty and precisely adhering to complex instructions.