Microsoft: Phi-3 Medium 128K Instruct

Microsoft's Phi-3 Medium 128K Instruct, released on May 23, 2024, demonstrates exceptional speed and cost efficiency, consistently ranking among the fastest and most competitively priced models across six benchmarks. This 14-billion parameter model is designed for advanced language understanding and reasoning, excelling in common sense, mathematics, logical reasoning, and code processing. While its overall performance on the provided benchmarks shows significant variability, it achieved a strong 76% accuracy in Reasoning, indicating a notable strength in complex problem-solving. However, the model exhibited very low or zero accuracy in Coding, Instruction Following, Ethics, and General Knowledge, suggesting these areas are significant weaknesses. Its 4% accuracy in Email Classification is also quite low. The model's description highlights its state-of-the-art performance among lightweight models at release, even approaching Llama3 70B levels on MMLU-Pro, which contrasts with some of the baseline benchmark results provided. Further investigation into the specific nature of these baseline tests versus the MMLU-Pro evaluation would be beneficial to reconcile these observations.