🔍 Read the full analysis: Z.ai’s Legal AI: What A 1-In-5 Benchmark Pass Rate Means on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria task pass rate on Vals AI’s version of Harvey’s Legal Agent Benchmark, placing seventh among 81 models in the archived leaderboard. The model passed 92.5% of individual criteria, but a task counts only when every required criterion passes; the score is not evidence that its legal work is ready to use without review.
Z.ai’s GLM-5.3-FlashX-Legal passed every required criterion on 19.17% of tasks in Vals AI’s archived evaluation of Harvey’s Legal Agent Benchmark, placing seventh among 81 models on that leaderboard. The score measures end-to-end task completion under strict all-or-nothing grading, not whether the model got 19.17% of legal questions right: Vals also reported a 92.5% pass rate on individual criteria.
Z.ai describes the model as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its developer documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning in what the company describes as tens of thousands of task environments modeled on real legal workflows. Z.ai recommends connecting the model to external services for laws, cases and company information.
The reported score comes from Vals AI’s public leaderboard, which uses Harvey’s benchmark and grading protocol. In the archived Version 1 table, GLM Legal scored 19.17%, ranked seventh of 81 models and cost $0.15 per test. The base model, GLM 5.3 Flash, scored 6.67% on that same table. The difference is 12.50 percentage points, matching Z.ai’s stated improvement over its base model. Z.ai’s pricing page lists separate token prices of $0.37 per million input tokens and $1.25 per million output tokens.
Harvey’s benchmark treats a task as complete only when every rubric criterion passes. It includes 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written pass/fail criteria. Vals says its leaderboard evaluation uses a private holdout of 120 tasks, disables internet access, and averages task pass rates from two AI judges. The 19.17% result therefore means the model cleared every criterion on roughly one in five tasks in that evaluation; on the rest, at least one criterion failed.
Why All-Criteria Passing Matters
The result illustrates a gap between partial accuracy and complete legal work. A model can satisfy most individual requirements yet still fail a task because one citation, deadline, recommendation or other criterion is wrong. Vals reported GLM Legal passed 92.5% of individual criteria, but that figure does not mean that 92.5% of its work products were complete or suitable for use.
For law firms and legal departments, the distinction affects how benchmark results should inform procurement and oversight. The leaderboard offers a comparative test under a defined setup, but it does not establish that the model is reliable for a particular client matter, practice area or jurisdiction. Any deployment would still require human verification of legal authorities, facts and deadlines, as well as a decision about how client information is handled.
The archived table also puts the result in perspective: GLM Legal ranked seventh, while the top listed model scored 25.42%. The benchmark is demanding, and performance across models was limited. Harvey’s own earlier baseline reported a best all-pass rate of 7.1% among the frontier models it tested, though that was a separate run and is not directly comparable to Vals’s result.
Products Worth Considering
Logitech MX Brio Ultra HD 4K Webcam, Graphite
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
FLEXISPOT Mochi-Pro Zero-Wall Oversized Power Recliner, Beige
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. Each task gives an agent an instruction and client-matter materials, then asks it to produce work for review. The rubric can test specific details such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey’s all-pass rule is intentionally strict: a work product that misses one required item is not counted as a completed task, even if it satisfies many others. Vals’s leaderboard applies that approach to a private holdout set, rather than simply reporting performance on the public task examples. Z.ai’s documentation gives the 19.2% figure, while its October 13 AutoClaw blog post identifies Vals as the independent evaluator. Vals listed the model’s release as October 12.
Z.ai calls the result state-of-the-art among open-source models in this evaluation. That wording is the company’s characterization; its cited documentation does not define the comparison set. The source material also does not establish that the legal model’s weights are publicly available. Its release and evaluation claims should not be taken as proof of open-source availability.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
Limits of the Benchmark Evidence
The leaderboard result is based on 120 private tasks and two AI judges, not a published assessment of every type of legal work. The available material does not show how performance varies across practice areas, jurisdictions, task difficulty or different client documents. It also does not establish how well the benchmark score predicts performance in live firm workflows.
There are other open questions for buyers. The source material does not specify who processes client data in a production deployment, what contractual or data-governance protections apply, or what human review Z.ai expects before a draft is used. A small spot check described by the source found legal rulings that could not be located in some GLM Legal citations; it involved only four tasks and was not a benchmark, so it cannot establish an overall error rate. The same spot check found different citation errors in two Claude models, underscoring that the anecdote is not a comparative evaluation.
Z.ai’s claim of state-of-the-art performance among open-source models also remains difficult to assess from the supplied material: the comparison group is not named, and published weights for the legal model were not identified. The company’s intended-use description and benchmark score do not settle whether particular outputs are accurate, legally appropriate or safe to rely on.
What Buyers Should Check Next
The next practical test is whether the model performs reliably on a firm’s own representative tasks, with its usual documents, jurisdictions and review procedures. Firms considering a trial can compare complete work products against their existing standards, track citation and deadline errors, and distinguish tasks that need extensive correction from those that save staff time.
Before connecting the system to client or company records, prospective users will also need details on data processing, retention and access controls, along with a clear policy for human review. Z.ai recommends connecting the model to external services for legal and company information, but the supplied documentation does not answer the procurement and governance questions that follow from such integrations.
The source material identifies no next benchmark release or further evaluation date. The immediate evidence remains the archived Vals result and Z.ai’s documentation; additional leaderboard runs, technical disclosures or customer testing could clarify how broadly the score applies.
Key Questions
What does the 19.17% score mean?
It means GLM Legal passed every required criterion on 19.17% of tasks in Vals AI’s evaluation. A task failed if it missed even one criterion, so the number is not the percentage of individual questions answered correctly.
How can the model have a 92.5% criteria pass rate but a 19.17% task pass rate?
Each task contains multiple criteria. The model passed 92.5% of criteria, according to Vals, but a task counted as complete only when all its criteria passed. A single failed item could make the entire task a failure.
Was the benchmark result independently evaluated?
Z.ai says the result was independently evaluated by Vals AI, and Vals’s archived leaderboard lists GLM Legal at 19.17%. Vals says its evaluation used a private holdout set and two AI judges.
Does the score show that GLM Legal is ready to give legal advice?
No. A benchmark ranking does not establish that a model is suitable for unsupervised legal work or any particular matter. The supplied material describes professional workflows and draft production, but human review and verification remain necessary questions for users.
Is Z.ai’s legal model open source?
The source material does not establish that the legal model’s weights are publicly available. Z.ai claims state-of-the-art performance among open-source models, but it does not name the comparison set in the cited documentation.
Source: ThorstenMeyerAI.com
