NewsBite analysis · AI, Data, Programming
VB
Qwen 3.8-Max, Opus 5 show why scores miss cost
The launch of Alibaba's Qwen 3.8-Max has stirred debate over AI model benchmarking, particularly concerning reliance on raw performance scores that overlook cost and budgeting. An independent evaluation suggests that time budgets can heavily influence results, calling for a shift towards a cost-per-success metric that accounts for all expenditures, including failed attempts. This would guide users in selecting models while accurately depicting true operational costs.
MonitorConfidence: MediumShort term horizon
Published 6 Aug 2026, 16:25 UTC5 min read
Visit original article ↗Lead image — VB
Summary
TL;DR
- Alibaba released the Qwen 3.8-Max model amid benchmark discussions.
- Cost per successful task metrics are proposed for better model evaluation.
- Token and time budgets heavily influence performance evaluations.
- The competitive landscape is shifting towards standardized cost metrics.
- Independent tests reveal discrepancies between model claims and actual performance.
Interesting facts
- Alibaba's Qwen 3.8-Max was ranked differently based on time and token budgets.
- Timeouts accounted for 79% of unresolved runs in certain benchmarks.
- The push for cost-per-success metrics is gathering traction in the industry.