Commerce Test Shows Where AI Agents Break Down

The potential for agentic artificial intelligence and its open issues are well-established. The promise of agentic commerce rests on the assumption that the agent gets the job done. So, Alibaba put the agents to work on real tasks and gave them a grade.

Alibaba.com released CommerceAgentBench, a public testing toolkit on GitHub, to measure how well AI can handle real online business tasks. The toolkit tested 13 AI model families on 107 real commerce tasks and graded each one pass or fail, according to results published on the CommerceAgentBench GitHub page. An agent got no credit for a well-written response. It passed only if the job was done correctly. A product listing had to go live with the right attributes. A shipment had to be booked on a route that exists.

Claude Opus 5 got the highest score of 61.7%, per the GitHub page.

Alibaba.com President Kuo Zhang wrote in a Sept. 9 Fortune commentary that the top score is “high enough to be useful and low enough to be a warning.”

The test covered procurement, logistics, product listing, fulfillment and after-sales service. Agents had to review hundreds of unstructured emails, spot payment fraud, calculate landed costs and book shipping routes across carriers, according to a Sept. 9 Alibaba press release. The test drew on tasks from 10 million active small business users, 1.6 million conversations and 200,000 execution traces. The figures are the company’s own. The tasks fell into seven categories and four levels of autonomy.

Agents worked inside local replicas of commerce software, including product publishing forms and freight booking tools. A task passed only when every verifier check passed, per the GitHub page.

Long Workflows Trip Up Agents

The failures clustered. Zhang wrote in his commentary that agents missed a payment anomaly buried in a supplier email thread of 300 messages. Landed cost broke down once several variables moved at once. After-sales disputes failed when documents disagreed with each other. Multi-leg shipping routes were hard every time.

Each of the jobs made an agent hold information across many steps. The top model, Claude Opus 5, averaged 63 tool calls and about 10 minutes per task, the GitHub page showed.

White-collar work produced a similar result. A benchmark from training data company Mercor called APEX-Agents found that the best models got fewer than a quarter of tasks right, TechCrunch reported in January. The tasks came from consulting, investment banking and law. Mercor CEO Brendan Foody said in the report that the biggest stumbling point was tracking down information across multiple domains.

The Harness Changes the Score

A model doesn’t work alone. It runs inside a harness that gives it tools and memory. The GitHub page listed 13 model families on three harnesses, and the top models scored differently on each. Claude Opus 5 passed 61.7% of tasks on Alibaba’s Accio harness, 60.7% on Pi and 56.1% on OpenClaw. Gemini 3 Flash ranked last and passed 29% on all three.

“No single model led across the board, reinforcing the case for task-level routing, the idea that different tasks are best handled by different AI models rather than relying on one general-purpose model for everything,” Alibaba said in its Sept. 9 press release.

Consumers and Merchants Set Their Own Limits

Demand is already arriving on the consumer side. The PYMNTS Intelligence report “Will the 2026 Shopping Season Go Agentic?” found in September that about 132 million adults in the United States have bought a retail product with AI’s help. Amazon captured 59% of these AI-assisted purchases.

Merchants are choosing what agents may touch. The PYMNTS Intelligence report “Global Digital Shopping Index: The Agentic Commerce Deep Dive” found in July that 46% of merchants named pricing as the function they’re least willing to hand to agents. Fraud, disputes and liability followed at 42%.

Alibaba.com said on the GitHub page that it will run outside models on request, including pre-release builds. It’s asking developers to contribute new test environments, which enter the scored set in a later version.

For all PYMNTS AI coverage, subscribe to the daily AI newsletter.