AI System Documentation and Model Cards

Objective

Maintain current, accessible technical documentation (model cards, datasheets, system cards) for every production AI system, sufficient to enable risk-based use, regulatory review, and downstream consumer understanding.

Applicability

AI types

LLM, Agentic AI, Traditional ML, Computer Vision, Generative AI, Multi-modal, Recommender

Deployment models

SaaS, Self-hosted, Hybrid, Edge, Embedded

Lifecycle stages

Deployment, Operation & Monitoring

Risk domains

Governance, Model

Regulatory regimes

EU AI Act, ISO 42001, NIST AI RMF

Company size

SMB, MidMarket, Enterprise

Rationale

EU AI Act Article 11 and Annex IV require technical documentation for high-risk systems. ISO/IEC 42001 clause 7.5 and NIST AI RMF MAP-4 require traceability and transparency. Model cards (Mitchell et al.) and datasheets (Gebru et al.) are established practices that translate technical specifications into reviewable artifacts for non-specialist stakeholders.

Control narrative

Every production AI system has a Model Card and Dataset Datasheet maintained alongside the model. Model Cards include: model details (developer, version, type, training data summary, evaluation data summary), intended use and out-of-scope use, factors (relevant demographic / environmental), metrics (performance, fairness, robustness), evaluation data, training data, quantitative analyses, ethical considerations, caveats and recommendations. Datasheets include: motivation, composition, collection process, preprocessing, uses, distribution, maintenance. Documentation is reviewed at material change and at minimum annually. For systems used by downstream business teams, a non-technical 'use card' is also published.

Test of Design

Procedures

1. Obtain the AI Documentation standard and confirm required artifacts (Model Card, Datasheet, optional Use Card).
2. Confirm content requirements align with Annex IV of the EU AI Act for high-risk systems and with widely adopted templates (Model Cards, Datasheets).
3. Confirm storage location is accessible to stakeholders who need it.
4. Confirm refresh requirements (material change, annual minimum).

Inquiries

• Who authors and reviews Model Cards?
• How is Annex IV technical documentation maintained for EU AI Act high-risk systems?
• How is documentation accessibility balanced with confidentiality (e.g., for systems whose details are competitively sensitive)?

Inspections

• AI Documentation standard.
• Model Card and Datasheet templates.
• Sample of completed Model Cards and Datasheets.
• Documentation review records.

Test of Operating Effectiveness

Procedures

1. From the inventory, sample production AI systems.
2. For each, confirm a Model Card and Datasheet exist and are current (reviewed within the last 12 months or since last material change).
3. Confirm content covers required sections substantively.
4. For EU AI Act high-risk systems, confirm Annex IV technical documentation completeness.
5. Confirm the documents are accessible to the documented stakeholders.
6. Trace 2 metrics in the Model Card to the underlying evaluation report and confirm consistency.

Sample-size guidance

Evidence requirements

Required

• AI Documentation standard PDF/Word · At fieldwork
• Model Cards for sampled systems PDF/Markdown · Per sample
• Datasheets for sampled systems PDF/Markdown · Per sample

Supporting

• Annex IV technical documentation for EU AI Act high-risk PDF/Word · Per sample

Retention: Lifetime of the AI system + 10 years (EU AI Act high-risk); + 7 years (regulated); + 3 years (otherwise)

Framework mappings

Related controls

• AI-CTRL-002

  Training Data Provenance and Lineage

• AI-CTRL-011

  Bias Testing and Fairness Validation

• AI-CTRL-017

  Pre-Production AI Evaluation Gates

References

• Mitchell et al. — Model Cards for Model Reporting
• Gebru et al. — Datasheets for Datasets
• EU AI Act Annex IV — Technical documentation