Most law firms buying AI tools have never seen a model card.

Most law firms buying AI tools have never seen a model card. That's like signing a retainer without reading the engagement letter. Model cards are standardized documents that ship with machine learning models, explaining what the model does, how it was tested, where it fails, and what data trained it. Google's research team introduced them in 2018, and the EU AI Act now effectively mandates them for high-risk systems.

If your firm is evaluating legal AI vendors — for research, document review, contract analysis, anything — you need to know how to read a model card and what red flags to look for. A vendor that won't produce one is a vendor hiding something. The 2025 AI Transparency Atlas framework weights safety evaluation at 25% and critical risk disclosure at 20% of a model card's transparency score. Those aren't nice-to-haves. They're the sections that tell you whether this tool belongs anywhere near client data.

What a Model Card Actually Contains

A model card covers recurring documentation categories: Model Architecture, Compute Requirements, Evaluation Metrics, License, Intended Use, Training Data, Limitations, Bias and Fairness, Safety Evaluation, Out-of-Scope Use, and Interpretability. Think of it as a nutrition label for AI. The "Intended Use" section tells you what the model was designed to do — and critically, what it wasn't designed to do. A model built for general consumer chat isn't validated for legal research, period. The "Training Data" section reveals whether the model was trained on legal corpora, public web scrapes, or proprietary datasets. The "Limitations" section is where vendors are supposed to disclose known failure modes. If that section is thin or missing, you're looking at a documentation gap that empirical studies on platforms like Hugging Face have found in thousands of published model cards. For legal AI, the most important sections are Intended Use (does it cover legal work?), Training Data (was legal data included and how?), Evaluation Metrics (tested against legal benchmarks?), and Limitations (known hallucination rates?).

Why Law Firms Should Demand Model Cards From Every Vendor

ABA Formal Opinion 512 requires lawyers to understand whether AI systems are "self-learning" and to obtain informed consent before using client data in AI tools. You can't satisfy that duty if you don't know what's under the hood. A model card is the minimum documentation that lets you answer basic due diligence questions: Was this model trained on data that could include privileged information? What are its known bias patterns? Has it been evaluated for accuracy in legal contexts? The EU AI Act's transparency requirements are already shaping what vendors must disclose, and firms with European clients or cross-border work can't ignore this. Sustainability Model Cards — a 2025 extension — now include environmental impact data like energy consumption and carbon emissions. Some clients, particularly institutional ones with ESG mandates, are starting to ask about this. Demanding model cards isn't being difficult. It's practicing competent vendor management under Rule 1.1's technology competence duty, which 40 states plus D.C. have now adopted.

How to Read a Model Card: A Managing Partner's Guide

You don't need a computer science degree. Start with five sections. Intended Use: If it says "general purpose" or "consumer applications" and the vendor is pitching it for legal research, that's a mismatch. Training Data: Look for specifics. "Trained on publicly available data" is vague. You want to know if legal texts, case law, or statutes were included. Evaluation Metrics: Look for benchmarks relevant to legal tasks — accuracy on citation verification, hallucination rates, performance on bar exam questions. Generic benchmarks like MMLU aren't enough. Limitations: This section should be substantive. If it's two sentences, the vendor either doesn't know or won't say. Neither is acceptable. Safety Evaluation: The 2025 transparency framework gives this section the highest weight at 25%. Look for red-teaming results, adversarial testing, and documented failure cases. A good model card will also include an "Out-of-Scope Use" section. If legal applications aren't listed as in-scope, ask the vendor to explain why they're selling it to law firms.

Red Flags That Should Kill a Vendor Deal

Here's what should stop a procurement conversation cold. No model card exists. If a vendor says they don't have one, they're either not serious about transparency or they're repackaging someone else's model without understanding it. The Limitations section is empty or generic. Every model has limitations. A vendor claiming otherwise is lying. No legal-specific evaluation data. If they can't show you accuracy rates on legal tasks, they haven't tested it for your use case. Training data is undisclosed. You need to know whether client-similar data was in the training set and whether there are opt-out mechanisms. No version history. Models get updated. You need to know when the model card was last revised and whether it reflects the current production model. The AI Transparency Atlas evaluated major foundation models in 2025 and found significant documentation gaps across the board, particularly in limitations, evaluation, and environmental sections. If the biggest players struggle with complete documentation, imagine what smaller legal AI vendors are shipping.

Building Model Card Review Into Your AI Governance Framework

Make model card review a formal step in your vendor evaluation process. Before any legal AI tool gets approved, require the vendor to produce a current model card or equivalent documentation. Assign review responsibility to your AI committee — ideally someone with enough technical literacy to assess whether the card is substantive or performative. Create a checklist based on the five critical sections. Score each vendor on completeness and specificity. Keep model cards on file and require updated versions when the vendor pushes model updates. This matters because a model card from six months ago may not reflect current capabilities or risks. For firms building internal tools or fine-tuning models, create your own model cards. Document what you built, how you tested it, and what it's approved for. This isn't just good practice — it's evidence of technological competence if your AI use ever gets scrutinized by a court or bar authority.

The Bottom Line: Model cards are the single best tool for evaluating whether a legal AI vendor is serious about transparency. They exist. They're standardized. And any vendor that won't produce one shouldn't get past your first screening call. Build model card review into your procurement process, train your AI committee to read them critically, and treat missing documentation the same way you'd treat a contract with blank terms — as a deal-breaker.