AionLabs: Aion-RP 1.0 (8B)

AionLabs: Aion-RP 1.0 (8B) demonstrates exceptional speed and competitive pricing, consistently ranking among the fastest and most cost-effective models across seven benchmarks. This fine-tuned base model, designed for natural and varied writing in roleplaying scenarios, shows a mixed performance across various benchmarks. Its primary strength lies in its intended application, having achieved the highest ranking in the character evaluation portion of the RPBench-Auto benchmark. However, its general performance on baseline tasks indicates significant areas for improvement. The model exhibits a very high propensity for hallucination, with only 2.0% accuracy in acknowledging uncertainty. Performance in Ethics (14.0%), Instruction Following (0.0%), and Reasoning (12.0%) is notably low, suggesting limitations in complex ethical judgment, precise instruction adherence, and abstract problem-solving. General Knowledge (58.5%) and Coding (47.0%) show moderate accuracy, while Email Classification (34.0%) also falls short. The model's design as a fine-tuned base model rather than an instruct model likely contributes to its struggles with tasks requiring explicit instruction following and factual recall.