Somewhere in a large financial institution right now, a person is sitting in front of a screen. She has been designated as the human reviewer responsible for overseeing a batch of AI-generated credit decisions. She has 847 of them to get through before end of day. Each one is a real person's application — a small business owner in Soweto, a domestic worker in Durban, a teacher in Polokwane — and each one has been assessed by a model she did not build, trained on data she has not seen, producing outputs she cannot fully explain. She clicks through them, one by one. And the institution, in its governance documents, calls this human oversight.
This is not a hypothetical. It is the operational reality of AI deployment in financial services, not just in South Africa, but in financial systems around the world. And it is the reason why 'human in the loop' — one of the most frequently used phrases in AI governance — has become, in many cases, a fiction.
I want to be careful here. I am not saying that financial institutions are acting in bad faith, or that no meaningful human oversight of AI exists. But as algorithms process increasing volumes of decisions across insurance underwriting, claims processing, and credit scoring, the structural capacity of humans to meaningfully review those decisions is breaking down. We are maintaining the architecture of oversight while steadily eroding its substance.
The illusion of enablement
The core problem lies in a concept known in human-computer interaction as 'automation bias' — the psychological tendency of human operators to trust the outputs of automated systems and ignore contrary evidence, even when the system is wrong.[1]
When an AI model reviews a loan application and scores it as 'high risk', the model processes thousands of variables in milliseconds. If a human reviewer is placed in the loop, what exactly are they supposed to do? They do not have the time to manually recalculate the variables. They usually do not have the technical expertise to interrogate the model's weights. If they override the model and the loan defaults, they must answer to their managers for ignoring the costly technology the bank just procured. If they simply click 'approve' on the model's recommendation and the loan defaults, the model takes the blame.
The rational, inevitable outcome of this dynamic is that the human reviewer becomes a rubber stamp. They are in the loop not to provide rigorous oversight, but to absorb the liability. This dynamic was explicitly acknowledged in the 2024 joint survey on AI adoption conducted by the FSCA and the Prudential Authority, where several institutions noted the challenge of maintaining genuine human autonomy in environments where algorithmic volume vastly outpaces human processing capacity.[2]
The legal mandate versus operational reality
Regulators internationally are beginning to recognise this gap between theoretical oversight and operational reality. The European Union's AI Act, which formally entered into force in 2024, explicitly mandates 'human oversight' for high-risk AI systems (which includes systems used to evaluate creditworthiness).[3] Crucially, Article 14 of the Act requires that human overseers must be able to properly understand the system's capacities, remain aware of automation bias, and be able to correctly interpret the system's output. They must also have the authority to override or reverse the system's decisions.
This sounds robust on paper. But as legal scholars have pointed out, there is a vast difference between giving an employee the theoretical authority to override an AI system, and giving them the practical tooling, time, and institutional cover to actually do so.[4]
Moving from a 'Loop' to a 'Guardrail'
If putting a human in the loop for every transaction is increasingly performative, how do we ensure accountability in an AI-driven financial system? The answer requires a shift from transaction-level oversight to systemic guardrails.
First, regulators and institutions need to stop pretending that humans can effectively supervise high-volume algorithmic decisions at the point of execution. The oversight must move upstream to the design, training, and testing phases of the model, and downstream to rigorous, continuous post-deployment auditing. The humans should be reviewing the system's parameters and aggregate outcomes, not clicking 'yes' on individual outputs they cannot possibly unpack.
Second, where human oversight at the transaction level is maintained, the institution must be required to prove that the oversight is structurally meaningful. If an institution claims a human reviews AI credit decisions, supervisors should ask: What is the override rate? If the reviewer agrees with the machine 99.9% of the time, the human is not overseeing the machine; the human is submitting to it. Furthermore, does the interface provide the human with 'explainable' reasons for the AI's decision in plain language? Does the reviewer have a quota that makes genuine review mathematically impossible?
Third, we need to rethink liability. The 'human in the loop' is often deployed as a liability shield — a way for the institution to say, "A person made this decision, not a computer." But if the person was structurally incapable of doing anything other than agreeing with the computer, the liability must rest with the institution that deployed the system, not the employee who clicked the button.
As South Africa integrates AI more deeply into its financial architecture, we must not let 'human in the loop' become an empty incantation. True oversight requires more than just a person in a chair. It requires transparency, time, and the institutional power to say no to the machine.