🔍 Read the full analysis: Amazon’s AI Bets Make The Case For Measuring More Than Tokens on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Amazon published engineering guidance on October 27 telling managers to judge AI use by its results, not by token counts or tool adoption alone. The advice comes as reports point to a pullback in some Nova models and Amazon’s filing values its Anthropic holdings at about $190 billion; the company’s third-quarter results are due October 29.
Amazon published new guidance for its software engineers on October 27, warning managers not to use token counts or other AI usage measures as performance targets. The memo says the relevant test is whether work improves outcomes for customers, the business and teams, a message that lands as Amazon’s own Nova models and its investment in Anthropic show different results across the company’s AI bets.
The memo, written by Terk Terkowitz, a vice president and distinguished engineer in Amazon’s stores business, sets out a revised role guideline for software engineers. Amazon says it was written by the company’s most senior engineers. It does not define the role around particular AI tools, arguing that such definitions could quickly become outdated. Instead, it anchors the role in impact on customers, the business and the team.
Because long-term outcomes can take time to measure, some teams have used proxies such as token usage, AI tool adoption, code changes and reviews. The memo calls these “deeply flawed metrics” when they are disconnected from results or used as targets. It says that approach can increase costs and system complexity. But it also says the measures currently “correlate well” with important outputs, and that teams adopting AI most heavily are delivering more. The memo’s distinction is between using metrics to learn and using them to score people.
It tells leaders to examine both low and high AI adoption: the 10th percentile to understand why some teams use tools less, and the 90th percentile to check whether heavy use leads to better delivery. Engineers whose work does not benefit from current tools should record why, the memo says. Leaders are also asked to use AI tools on their own production work and explain where they helped and where they did not.
Six months at Amazon
From token leaderboards to an outcomes memo
Amazon spent 2026 learning the same lesson twice: once with how its engineers use AI, once with its own AI models.
- May 2026Token leaderboards backfireFinancial Times (via The Decoder): staff give an internal agent tool pointless tasks to inflate token counts. Reported, anonymous sources.
- Late May 2026KiroRank “deprecated”Amazon spokesperson to Business Insider. SVP Dave Treadwell: “Please don’t use AI just for the sake of using AI.”
- 24 Jul 2026San Francisco AGI Lab site closesConfirmed by Amazon to GeekWire. Frontier model research continues under Pieter Abbeel.
- 28 Jul 2026Most flagship Nova models wound downBusiness Insider, people familiar: Premier, Omni, Reel, Canvas to “keep the lights on”. Amazon: models “remain one of the most important things we’re working on”.
- 14 / 30 Sep 2026Nova Sonic v1, Nova Reel, Nova Canvas reach end of lifeAWS Bedrock model lifecycle page. Requests to them now fail.
- 26 Oct 2026Anthropic files its S-1Amazon notes: $6.2 billion principal, 12% ownership cap that Amazon can waive. Anthropic commits at least $110 billion to Amazon through April 2036.
- 27 Oct 2026The memo: outcomes, not tokenmaxxingUsage and throughput metrics are “deeply flawed” as targets. “Tokenmaxxing is at odds with frugality.”
- 29 Oct 2026Amazon’s third-quarter resultsAnnounced by Amazon. Not yet published when this article went live.
Sources: Amazon engineering memo (27 Oct 2026); Business Insider (29 May and 28 Jul 2026); GeekWire (24 Jul 2026); The Decoder (12 May 2026, summarising the Financial Times); AWS Bedrock model lifecycle (read 28 Oct 2026); Anthropic S-1 (26 Oct 2026).
Powered by Thorsten Meyer AI
Amazon’s AI Scorecard Shifts to Results
The guidance matters beyond engineering reviews because usage targets can reward activity even when it adds little value. A token leaderboard may encourage employees to generate more AI output without showing that customers receive better service or teams complete useful work faster. Amazon’s stated concern is practical as well as managerial: usage targets can raise costs and complexity.
The company’s own AI portfolio offers a relevant, though incomplete, illustration. Business Insider reported in July that most flagship Nova models had been moved to “keep the lights on.” Separately, AWS’s model lifecycle page lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. On the other side, Amazon’s June 30 filing valued its Anthropic holdings at about $190 billion, compared with about $18 billion invested, according to the source material. Those figures are reported valuations, not a measure of customer impact or realized cash returns. Amazon is due to report third-quarter results on October 29; those results may add financial context, but the memo does not supply a performance score for its AI strategy.
Products Worth Considering
Logitech MX Master 4 for Mac, Ergonomic Bluetooth Mouse with Haptic Feedback, Ultra-Fast Scrolling, USB-C Charging, for MacOS, iPadOS, Windo
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
As an affiliate, we earn on qualifying purchases.
From Token Leaderboards to Guidance
The memo follows reports of internal pressure around AI adoption. In May, the Financial Times reported that Amazon employees were assigning unnecessary jobs to an internal agent tool called MeshClaw to increase token counts. The report also described a weekly AI-use target and internal leaderboards. Those accounts were attributed to employees and remain reported claims in the supplied material, rather than facts independently established by the memo.
At the end of May, an Amazon spokesperson confirmed to Business Insider that a leaderboard called KiroRank had been “deprecated.” The company described it as an informal tracker created by a group of employees and said it was never intended to promote AI use for its own sake. The October 27 memo makes the company’s position more formal by placing the emphasis on outcomes in its engineering role guidance.
Amazon’s model portfolio has also changed. Business Insider reported in July that most flagship Nova models were shifted to maintenance. AWS’s lifecycle listings separately show retirements for Nova Canvas, Nova Reel and the first Nova Sonic. The material provided does not establish that the memo caused those product decisions, or that they resulted from a single assessment of model performance.
“KiroRank “was never intended to promote the use of AI for usage’s sake.””
— Amazon spokesperson, as quoted by Business Insider in May
What the Metrics Cannot Show
The memo does not give figures for how much AI use has changed engineering output, costs or customer outcomes. It says adoption metrics correlate with important outputs today, but does not quantify that relationship or say how it varies by team or task. It also does not set out a replacement scoring formula; it calls for leaders to use judgment and investigate differences between light and heavy use.
The basis and timing for the roughly $190 billion Anthropic valuation in Amazon’s filing are described in the supplied material, but the figure alone does not show realized gains or the investment’s eventual return. The relationship between the Nova product changes and Amazon’s broader AI strategy is also not established. Amazon’s third-quarter report is scheduled for October 29, but no results are available in this source material.
Amazon’s Results Due October 29
Amazon is scheduled to report third-quarter results on October 29, one day after the memo’s publication. The report may provide additional financial information about the period, though the source material does not say whether Amazon will disclose specific AI usage, model performance or customer outcome measures.
For engineering teams, the immediate next step in the memo is to examine why adoption differs across teams and whether heavier use improves delivery. Amazon’s guidance calls for examples from leaders’ own production work, including tasks where AI helped and tasks where it did not. How the company will assess those findings over time remains unstated.
Key Questions
What did Amazon’s October 27 memo say about AI use?
It said managers should focus on work’s impact on customers, the business and teams, and should not use token counts or other adoption measures as performance targets.
Does Amazon say engineers should stop using AI tools?
No. The memo says AI adoption metrics currently correlate with important outputs and asks leaders to examine adoption at both ends. Its warning is against treating usage itself as success.
What happened to Amazon’s KiroRank leaderboard?
Amazon confirmed to Business Insider at the end of May that KiroRank had been “deprecated.” The company described it as an informal tracker made by employees and said it was not intended to encourage AI use for its own sake.
What does Amazon’s filing say about its Anthropic holdings?
The source material says Amazon’s June 30 filing valued its Anthropic holdings at about $190 billion, after an investment of about $18 billion. That reported valuation does not by itself establish realized returns or future performance.
When will Amazon report its third-quarter results?
Amazon is scheduled to report third-quarter results on October 29, 2026. The supplied material does not include those results.
Source: ThorstenMeyerAI.com
