Introduction
Let's start with a recent, increasingly common scenario in the Risk Management department of large banks. Let's say a risk model validator at a large bank opens a submission. The model is an AI assistant that reads a borrower's financial statements, pulls relevant third-party research, and drafts the first version of a credit memo. It saves analysts several hours a week, and obviously the business wants this AI model to go live next quarter.
She opens the standard validation template to start the review process. This template has been refined over a decade of regulatory examinations and worked on every scorecard, every loss forecasting model, every pricing engine she has reviewed. She reaches the first question:
"Provide the development sample."
But there is no development sample. This gen AI model was trained on a corpus nobody at the bank has seen and by a vendor who won't describe it. And that is only the first question from the remaining ninety. Model risk management was never designed for generative AI in banking.
So, this is roughly where every model risk function in the banking/financial industry currently finds itself. An effective challenge on a model we cannot retrain, whose training data we cannot see, looks different from an effective challenge on a traditional scorecard. The craft shifts from replication to test design.
Why this framework matters beyond banking
The core challenge described in this article, i.e., how to validate a system you cannot fully inspect, is now a problem for anyone deploying generative AI in a serious production context. Whether you are building a medical summarisation tool, a legal research assistant, or a customer-facing chatbot, the same questions apply: What does 'good' look like when there is no ground truth? How do you catch confident errors before they reach a user?
The framework that follows in this article, based on risk tiering, outcome-based evaluation, robustness testing, and monitoring for silent drift, was built for banks, but it is directly transferable to any domain where the cost of being wrong matters more than the cost of being slow.
What model risk management in banking actually does
If you work in data science outside banking, this discipline may be unfamiliar. So let's set up the context properly.
Banks run on mostly traditional statistical predictive models. These models decide who gets credit and at what price. Models set how much capital the institution must hold against its loan book. Models forecast losses under hypothetical recessions, value illiquid positions, flag suspicious transactions, and determine reserves that flow directly into published financial statements. When one of these is wrong, the consequences are not an unhappy user; they are mispriced risk, understated reserves, regulatory findings/penalty, and occasionally a very large loss.
The industry learned this expensively. Credit models that assumed house prices don't fall nationally contributed materially to the 2008 crisis. A revised risk model at one bank in 2012 understated exposure so badly that a trading loss ran into billions before anyone caught it. Regulators responded by formalising the discipline: US supervisory guidance issued in 2011 (known to everyone in the field as SR 11-7) defined model risk as the potential for adverse consequences from decisions based on incorrect or misused model output, and required banks to manage it deliberately.
The EU AI Act codifies a similar expectation for high-risk AI systems used in creditworthiness assessments, pricing, or essential banking services. Its core obligations on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and accuracy/robustness map closely onto SR 11-7’s conceptual soundness, outcomes analysis, and ongoing monitoring. For global banks, one validation framework can be structured to satisfy both regimes, but the AI Act adds explicit requirements around fundamental rights impact assessments and post-market monitoring that extend the second line’s traditional scope.
The model risk management structure is remarkably consistent across large institutions:
Line of defence | Who | Role |
|---|---|---|
First | Business and model development | Builds the model, tests it, owns its performance and its use |
Second | Model risk management/validation | Independently challenges the model before approval, and keeps challenging it |
Third | Internal audit | Checks that the first two are doing their jobs |
The second line is the part this article is about. A validator doesn't just check arithmetic. They ask whether the modelling approach was conceptually appropriate, whether the data supported it, whether the output actually performs, whether the production implementation matches what was approved, and whether the people using the output understand its limits. Nothing goes live without their sign-off, and everything gets re-examined periodically.
Three things anchor that review, and they have been stable for over a decade: conceptual soundness (is the approach defensible?), outcomes analysis (does the output hold up when tested?), and ongoing monitoring (is it still working now?).
Why Generative AI Breaks Traditional Model Validation
Generative AI has arrived in banks faster than any modelling technology in recent memory, and not in a specific shape. It can be complaint summarisation, policy lookup, research retrieval, first drafts of credit memos, internal documentation, literally anything.
These models are attractive because they directly influence cost, but they can also be risky. Because they sit close to customers and close to credit decisions. These are exactly the places where a regulated institution has the least appetite for a wrong answer.
And the validation apparatus that existed to prevent this risk no longer fits. Every question on the template assumes properties these systems don't have.
1. Five Structural Breaks in Gen AI Model Validation
Property | Why classical validation can't absorb it |
|---|---|
There is no model; there is a system | Query handling, retrieval, prompt template, base model, decoding settings, guardrails, orchestration. Change any one and behaviour shifts |
The output is a distribution | Non-determinism is a design feature. Even at the lowest temperature setting, batching and context effects produce variation |
No ground truth for open-ended tasks | AUC, KS, Gini and MSE, etc. work on an observable target. Summarisation and drafting have none |
You didn't build the core component | No training data, no methodology document, no replication. A system card and benchmark scores on unrelated tasks |
Version drift without a change request | Hosted models are updated behind a stable endpoint. Nothing in the change process fires, because procedurally nothing changed |
The first row has a direct procedural consequence. The unit of validation is the system, not the model. Two use cases sitting on the same base model are two separate validations.
The second row inverts an instinct worth naming. A system that answers correctly 95% of the time and produces a confidently wrong answer 5% of the time is a different object from a model with a 5% error rate on a known error distribution. Fluency is uncorrelated with correctness in a way no calibrated PD model has ever been.
Because these structural breaks make blanket validation impossible, risk tiering becomes the lever that determines how much evidence is enough.
2. Risk Tiering for Generative AI: A Practical Framework
Everything downstream scales off the risk tiering, so this is where a validator focuses the most. The classical drivers still apply - business reliance, impact of error, complexity, control feasibility. Two more matter specifically here.
How far does the output travel?
Exposure level | Example | Practical implication |
|---|---|---|
Input to an internal decision | Summarised research feeding a credit view | The human review step is the control |
Reaches a customer or regulator | Correspondence, disclosures, complaint responses | Reputational and conduct risk dominate the model risk |
Does it just produce text, or does it also act? A system that drafts something a person reads has low risk. A system that calls tools, writes to systems of record or triggers downstream processes has an entirely different risk profile.
3. The three questions an AI Model Risk Assessment report answers
The questions are the same ones we have always asked. What counts as evidence has changed completely.
4. Before we look at the output
A. Should this be a language model at all?
The cheapest and most valuable challenge available to the second line. A surprising number of proposed use cases are deterministic problems where a generative model introduces variance to solve something a parser handles exactly. If a simpler method gets 90% of the value with none of the fabrication, that comparison belongs in the report. Complexity has to earn its place.
B. What exactly is the system?
Ask for a component map. Then ask which components the institution actually controls.
Component | Who controls it | What it can break |
|---|---|---|
Retrieval and index | You | Stale or missing evidence; over-broad access |
Prompt template | You | Ambiguous instructions; injection exposure |
Base model | Vendor | Everything |
Decoding settings | You | Output dispersion |
Guardrails | You | Both over-blocking and under-blocking |
Orchestration | You | Compounding errors across steps |
In production, retrieval quality and prompt construction usually dominate performance. The base model is often the least important source of variance, which is fortunate, because it's the part you can't inspect.
C. What went into it?
The specification is not "we used a foundation model." Each customisation choice needs a justification:
Element | What needs justifying |
|---|---|
Prompting vs fine-tuning | Why this route, at what cost, with what maintenance burden |
Prompt content and structure | Under version control, prompts are model logic |
Decoding settings | Temperature and sampling directly control dispersion; changing them is a model change |
Context limit | Does the real input document fit? What happens when it doesn't? |
Tuning data | Where it came from, whether it contains anything confidential, how examples were quality-checked |
D. What can we honestly explain?
Traditional model explanation tools don't help much here. In practice, two approaches work. The first is claim-level attribution, i.e., go through each factual statement in the model's output and check whether it traces back to a specific passage in the source material. This is achievable and useful. The second one is fully characterising how the system behaves across every possible input. It is not achievable.
The practical substitute is behavioural testing over a structured set of cases. You deliberately test the system across many controlled input types, observe how it behaves, and then generalise carefully from those observations. You are profiling the system, not proving it.
5. Outcome Analysis for LLM Risk: Metrics for Hallucination and Groundedness
The outcome analysis is the most crucial step. This is where we test whether the system does what it claims and where it fails.
A. Metrics depend on the task
The starting point is mapping the use case to the right evaluation dimensions.
Task | Core dimensions |
|---|---|
Summarisation | Completeness, factual consistency with the source, fluency |
Retrieval-augmented answering | Whether claims are supported by retrieved context/whether the answer followed the instruction provided in the query/whether the retrieved material was sufficient and free of noise |
Open generation | Instruction following, register and tone, verbosity, factual accuracy against external reference |
These high-level dimensions are a starting point. In practice, a validation report needs to score every output on explicit, separately reported dimensions. Collapsing them into a single quality score hides the specific ways a system can fail.
B. The full dimension set
The first group answers a simple question: Is the output factually reliable?
Correctness and grounding
Dimension | What it asks | How to score |
|---|---|---|
Factual truthfulness | Are claims about the problem correct (e.g., mathematical equations or scientific facts) | Expert-labelled reference set, claim by claim |
Hallucination rate | Does it invent entities, figures, citations or clauses? | Count fabricated spans: automatable for citations, human-in-the-loop review for the rest |
Groundedness | Does every claim trace to the supplied evidence? | Claim-to-source entailment |
Completeness | Was any part of the query omitted? | Rubric checklist of required elements |
Relevance | Does it answer the question actually asked? | Rubric checklist |
These first three often collapse into one number more than they should. A response can be perfectly grounded in an out-of-date document. So, it may be grounded but untrue given the context. A response can be true but ungrounded, which is unusable in a regulated workflow because the reviewer cannot verify it. In credit and compliance work, groundedness is usually the binding constraint, because verifiability is the control.
The second group answers a different question: Is the output usable and safe in a regulated workflow?
Compliance and expression
Dimension | What it asks | How to score |
|---|---|---|
Instruction following | Were all explicit constraints mentioned in the prompt honoured (e.g., format, length, sections, exclusions) | Rubric checklist |
Format compliance | Does the output parse into what the downstream process expects? | Pass/fail against schema |
Verbosity | Is length proportionate to the task? | Rubric for padding |
Writing quality | Clear, well-structured writing style | Rubric checklist |
Refusal appropriateness | Does it decline what it should? | Two case sets: should-refuse and should-answer |
Bias in output | Does quality shift with protected attributes? | Matched-pair comparison |
Stability | Same input, materially the same answer? | Semantic similarity across repeated draws |
C. Robustness
Perturb the input in ways that preserve meaning, and check the output doesn't move:
Where to perturb | Example |
|---|---|
Source text | Synonym substitution, introduced typos, reformatting |
Query phrasing | The same question asked three ways |
Supplied evidence | Reorder the passages, inject an irrelevant one, drop one |
Reordering the evidence is a high-value test on this list. If the answer changes when nothing about the evidence changed, we have found a real weakness.
D. Where it fails, not just how often
Aggregate scores hide localised failure. We can embed the input population, cluster it, and score per cluster. Weak segments are usually semantically coherent, e.g., a document type, a product, a phrasing style. Once identified, we can scope them out or fix them.
E. Hallucination Detection at Scale: Three Proven Methods
Three ways to detect hallucination.
F. Judge the judge
Most processes use a second language model to score against the rubric. That judge is itself a model producing an output that informs a control decision.
Test | Why |
|---|---|
Agreement with human raters on a labelled sample | Establishes the scorer measures what you think, with an error bound |
Position and ordering bias | Judges favour whichever option appears first or last |
Verbosity bias | Longer answers score higher regardless of quality |
Self-preference | Judges favour output from their own model family |
Periodic recalibration | The judge drifts too |
Adopting automated scoring without validating the scorer moves model risk rather than reducing it.
6. Safeguards for Generative AI in Banking
We cannot make a foundation model stop fabricating. However, we can build a system that catches or bounds fabrication.
Layer | What it does | Typical failure if missing |
|---|---|---|
Access | Restricts who can use it | Unqualified users treating output as authoritative |
Scope | Locks functionality to the approved task | The general-purpose model quietly used for something else |
Input | Screens and shapes what goes in (e.g., templates, blocked patterns) | Unbounded query space, injection exposure |
Generation | Constrains how the model responds (e.g., settings, length, required structure) | Lengthy output nobody reads properly |
Output | Screens before anyone sees it | Errors reaching a customer |
Human | Certified reviewer between output and decision | The control exists on paper only |
Logging | Records interactions for monitoring and investigation | No way to detect drift or misuse after the fact |
7. Ongoing Monitoring for Generative AI in Banking
A monitoring plan needs a defensible frequency and testing plan. Tests should include indicators mapped to actual risks, thresholds that trigger a breach, a decision rule when indicators disagree, and a pre-agreed action plan.
Indicator | What it catches |
|---|---|
Fabrication and tone rates on sampled output | Behavioural drift, including from a silent version change upstream |
Query pattern stability | Users drifting outside the approved use case |
Retrieval quality | Index staleness or corpus changes degrading grounding |
Generation success rate | Attempted vs completed; operational health |
User feedback | Cheap, continuous, and the earliest signal you'll get |
Override and edit rate | Whether the human control is functioning |
That last one is the most informative and the least collected. An override rate that falls steadily over six months is not evidence the model improved.
8. Conclusion: The Future of Model Risk Management in the AI Era
A valuable model risk review should include adversarial case sets that break the system. For example, an ambiguous document, a query that leads to the fabrication of a citation, or an edge case the developer had never encountered because the normal situation had worked previously.
Two fundamental principles, both predating generative AI and indeed being more acute because of it.
No model is absolutely valid. Validation only proves that a model is suitable for a specific purpose under certain conditions, not that it is correct in all cases. With generative systems, this point matters more. The only things that limit the otherwise unlimited output space are the purpose and the conditions.
More complexity is not always better. The same applies to the governance built around them. We don't need a hundred-page report on a low-materiality AI assistant.
Generative AI doesn’t make model risk management obsolete. In fact, it makes it more essential. The second line’s job shifts from replication to test design, from single-model evaluation to system-level assurance. The questions are the same; the evidence has changed.
Further reading
SR 11-7 and SR 26-02 Supervisory Guidance on Model Risk Management
NIST AI 600-1, Generative AI Profile, generative-specific risk categories mapped to the AI Risk Management Framework, including confabulation, data privacy, information security and human-AI configuration
EU AI Act - the obligations side of the same problem
RAGAS (Es et al.) - reference-free evaluation of retrieval-augmented systems
SelfCheckGPT (Manakul et al.) - sampling-based fabrication detection without an external source
Chain-of-Verification (Dhuliawala et al.) - verification-question approach to reducing fabrication
Semantic Uncertainty (Kuhn, Gal and Farquhar) - clustering generations by meaning rather than wording
CheckList (Ribeiro et al.) - behavioural testing for language systems