OpenAI: o1

OpenAI's o1 model, created on December 17, 2024, is designed for extensive "thinking time" and optimized for STEM tasks using large-scale reinforcement learning. It exhibits moderate speed performance, ranking in the 24th percentile across benchmarks. Pricing for o1 tends to be at premium levels, placing it in the 1st percentile. A significant strength is its exceptional reliability, demonstrating a 100% success rate across all evaluated benchmarks. The model shows strong performance in several key areas. It achieved near-perfect accuracy in General Knowledge (99.8%) and Reasoning (100%), with the latter also noted for being the most accurate at its price point and among models of similar speed. Coding (92.0%) and Hallucinations (98.0%) also demonstrate high accuracy. While optimized for STEM, its Mathematics accuracy (87.0%) is solid, though not its highest percentile ranking. Ethics (98.0%) and Email Classification (98.0%) show consistent high accuracy. Instruction Following, while achieving 77.0% accuracy, still places it in the 89th percentile, indicating strong capability in this complex area. The primary weakness appears to be its premium pricing across all benchmarks.