The Model That Nobody Told

In an earlier piece on this blog, I argued that the "human in the loop" is frequently a rubber stamp — a person with neither the time nor the tooling to meaningfully interrogate a model's individual outputs.^[1] That argument focused on a single transaction: one loan application, one reviewer, one click. This piece is about a problem that a diligent, well-resourced human reviewer would still miss, because it is not visible at the level of any individual decision at all. It is only visible in aggregate, over time. It is called model drift, and it is arguably a bigger governance blind spot than the rubber-stamp problem, because almost nobody outside a model risk team is looking for it.

Two kinds of drift

Model risk practitioners generally distinguish between two related phenomena. Data drift occurs when the statistical properties of the incoming applicant population change relative to the data the model was trained on — a shift in income distributions, employment patterns, or the mix of first-time versus repeat borrowers, for instance. Concept drift is subtler still: it occurs when the underlying relationship between the input variables and the outcome the model is predicting changes, even if the applicant population looks similar on paper. A rise in interest rates, a change in labour market conditions, or a shift in consumer indebtedness can all quietly sever the statistical relationships a credit model learned to rely on.^[2]

Neither of these produces an error message. The model keeps running, keeps scoring, and keeps returning confident numbers. Nothing in the system tells anyone that those numbers are becoming less reliable, or that they are becoming less reliable for some groups of applicants faster than others.

Why this compounds the oversight problem

This is precisely why treating human review as a purely transaction-level control is structurally inadequate, whatever governance documents may claim. A reviewer assessing one application at a time has no visibility into the shape of the population the model is now scoring, and no baseline against which to notice that the shape has changed. Detecting drift requires aggregate monitoring — tracking accuracy, calibration, and fairness metrics like disparate impact ratios across a rolling window of decisions — not individual case review.^[3] International standard-setters have increasingly framed this as a model risk management discipline in its own right, distinct from, and prior to, transaction-level controls.^[4]

There is also a quieter, more troubling implication. If drift is degrading a model's accuracy and fairness, an attentive human reviewer's override rate should, in theory, start to rise — they should be catching and correcting more of the model's now-worse decisions. If the override rate stays flat while the underlying model quality deteriorates, that flatness is itself evidence that no genuine oversight is occurring. The reviewer is not becoming a better check on a worsening model; they are simply continuing to agree with it, because they have no independent way of knowing it has changed.

What supervisors and institutions should be asking

The practical response is not to abandon human review, but to relocate where it does its work. Institutions should be able to demonstrate, on an ongoing basis, that they are monitoring for both data and concept drift against a defined tolerance band, that breaches trigger a defined escalation and retraining process, and that fairness metrics are tracked by demographic group rather than in aggregate alone — since, as the interactive explainer below illustrates, a model can drift substantially worse for one group while looking broadly stable overall.

Below is a small interactive tool that lets you move a credit scoring model forward in time and watch exactly this dynamic unfold: accuracy degrading, the gap between applicant groups widening, and the human override rate barely moving throughout. It is a simplified simulation, not a real institution's data, but the shape of the problem is drawn directly from how these systems actually behave once deployed.