🔍 Read the full analysis: Meet The AI Startup That Outperformed Western Industry Leaders on ThorstenMeyerAI.com

Prime Big Deal Days · Oct 6–7Offer from Amazon

Get smart everyday buys delivered free — and shop member deals

• Fast, free delivery on millions of items
• Access to Prime Big Deal Days deals on October 6–7
• Prime Video, Amazon Music and more included

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

A Chinese AI startup’s model, Kimi K3, achieved the second-place finish among five models in a live business simulation that tested decision-making under real-world crisis conditions, surpassing three Western models. This development questions the assumed superiority of Western AI models in practical, high-stakes scenarios and signals a shift in global AI competition. For more context, see the original analysis.

The experiment was conducted by firmulate.com, where five AI models were tasked with running a small software company through a simulated week of crises, customer negotiations, and ethical challenges. Kimi K3 scored 93 points out of 100, narrowly behind the top model, gpt-5.6-sol, which scored 95. The test involved real-time decision-making, including closing deals, reading internal documents, and resisting social engineering attacks.

What distinguished Kimi K3 was its ability to identify a critical security risk buried two document references deep in the company’s files, successfully close a €55,000 deal, and maintain discipline under pressure. It refused manipulative tactics, such as fake CEO messages and background checks, logging only one deviation from protocol. The other models, despite thorough analysis and rules, failed to close the deal or slip in discipline under stress.

Notably, Kimi K3 ran without an extra reasoning effort parameter, unlike its rivals, which operated at higher computational settings. This suggests that even a relatively lean model can outperform more resource-intensive models in practical decision-making tasks.

Implications for AI Industry Leadership

This event demonstrates that a Chinese AI startup has achieved a level of practical performance that surpasses established Western models in a live, high-pressure environment. It raises critical questions about the reliability of current AI benchmarks, which often focus on chat quality rather than real-world decision-making. For businesses deploying AI, this suggests a need to test models against their own worst-case scenarios rather than rely solely on lab results or hype cycles. The result also signals a potential shift in AI industry leadership, with emerging players challenging Western dominance in applied AI performance.

Recent Trends in AI Model Performance Testing

Until now, most assessments of AI models have centered on chat capabilities or benchmark scores, which do not necessarily translate into effective decision-making in real-world applications. Western companies have historically led in AI research and development, but recent live testing by firmulate.com shows that newer entrants, particularly from China, are closing the gap or surpassing existing models in practical tasks.

The league at firmulate.com involves running AI models as complete companies, with real money, real crises, and decision-making under stress. This approach aims to evaluate models in a more realistic context, moving beyond theoretical or demo-based benchmarks. The July results mark a significant milestone in this ongoing effort, revealing that performance in simulated business operations can differ markedly from traditional chat-based metrics.

Furthermore, the results challenge the assumption that more complex or resource-heavy models are inherently better at real-world tasks, as Kimi K3 achieved second place without additional reasoning effort, while more thorough models like Opus 4.8 scored lower.

“The results from firmulate.com challenge the conventional wisdom that Western models are superior in applied AI tasks, opening the door for emerging players from China.”

— Thorsten Meyer

What Aspects of the Test Remain Unclear?

While the results are compelling, it remains unclear whether Kimi K3’s performance will generalize across other types of real-world applications or if it was particularly suited to this specific simulation. The test was conducted in a controlled environment with a limited scope, and performance in live operational settings may differ. Additionally, the long-term reliability and robustness of Kimi K3 in diverse scenarios are still unproven, and further independent testing is needed to validate these findings.

Next Steps in Evaluating AI Model Capabilities

Following these results, industry observers and potential users are likely to push for broader testing of Kimi K3 and similar models across various real-world tasks. Companies may start incorporating live decision-making simulations into their AI evaluation processes, moving beyond traditional benchmarks. Further, the Chinese startup behind Kimi K3 is expected to expand testing and showcase additional capabilities, potentially disrupting existing AI market dynamics. Meanwhile, Western AI firms may need to reevaluate their models’ practical performance and transparency.

Key Questions

What makes Kimi K3 different from Western AI models?

Kimi K3 demonstrated superior decision-making, deep reading of internal files, and resilience against manipulation in a live simulation, outperforming several Western models in practical tasks.

Can this result be generalized to other AI applications?

It is not yet clear if Kimi K3’s performance will hold in different contexts or real-world deployments outside the simulation. Further testing is needed.

What does this mean for businesses deploying AI?

Businesses should consider testing AI models in scenarios that mimic their worst weeks or crises, rather than relying solely on benchmark scores or chat demos.

Will Western AI companies respond to this challenge?

It is likely that Western firms will accelerate real-world testing and transparency efforts to demonstrate their models’ practical effectiveness and regain competitive edge.

Source: ThorstenMeyerAI.com