Can regulators trust black-box algorithms to enforce financial fairness?

As AI quietly reshapes the financial system, regulators face a pressing question; can they trust black-box algorithms to make fair decisions? These opaque models promise speed and objectivity, but what happens when no one understands how they work?

According to Luke DiRollo, CEO of ALMIS International, as banking regulation grows in complexity, the concept of risk is becoming both overused and underdefined.

He said, “Financial institutions are expected to adhere to a host of supervisory requirements, from capital adequacy and liquidity reporting to interest rate risk measurement. Yet how these requirements are operationalised – often through complex, black-box models – can lead to considerable ambiguity. The challenge becomes: how can regulators verify that the outputs of these models are fair, accurate, and comparable across institutions?”

DiRollo continued that fairness in this context isn’t just about equity in outcomes, its about consistency in interpretation and execution.

He said, “Unfortunately, the current regulatory paradigm, anchored predominantly in rules-based regulation, is failing to deliver this.”

Regulators, in their attempt to maintain oversight and comparability, often opt for rules-based regulation, said DiRollo. These are prescriptive, detailed requirements intended to eliminate ambiguity. However, this approach unintentionally creates a disproportionate burden on smaller institutions, he continued,

DiRollo said, “Each bank must effectively build its own data architecture to interpret and implement regulatory requirements. For instance, calculating Risk-Weighted Assets (RWAs) demands banks to collate data across a myriad of systems, map this data into a bespoke regulatory model, apply overlays and assumptions to reflect the intent of the rule and interpret evolving guidance and submit reports accordingly.”

He mentioned that this process is resource-intensive and heavily interpretative. “It leads to a scenario where the same rule yields different outputs depending on the institution’s internal systems, data quality, and modelling approach. The result? Spurious consistency. Reports look compliant on the surface but lack true comparability or meaningful insight,” DiRollo stated.

For the ALMIS CEO, this fragmented implementation underlines the notion of fairness in two critical ways – first around interpretive divergence. “With no common data model or processing architecture, each firm produces outputs based on its own assumptions. This leads to wide variations in reported metrics like capital ratios or liquidity buffers, even when the underlying risk profiles are similar.”

Secondly around regulatory arbitrage. In this area, larger institutions with more sophisticated modelling capabilities can structure their portfolios or data in ways that reduce regulatory burdens without a corresponding reduction in actual risk. “The implication is stark: the fairness that regulators seek to enforce is undermined by the very framework designed to ensure it,” said DiRollo.

While institutions pour effort into interpreting rules and submitting reports, the focus drifts from identifying and managing real risks. In practice, compliance becomes a proxy for safety – a dangerous assumption, in the words of DiRollo.

He explained, “This is where the Peltzman Effect, a concept from economics, becomes pertinent. It suggests that individuals (or institutions) adjust their behaviour in response to perceived safety. In banking, a similar dynamic plays out: the more a bank believes it has satisfied its regulatory obligations, the more likely it is to underweight emerging or non-prescribed risks. The illusion of compliance fosters complacency.

Another overlooked consequence of the current regulatory model for DiRollo is the immense overhead that it can impose on institutions, particularly smaller banks and building societies.

“Compliance with rules-based regulation requires dedicated teams of analysts, systems developers, and risk professionals whose primary function becomes interpreting and applying regulation, rather than optimising the bank’s performance or understanding its balance sheet,” said DiRollo.

He went on, “This is compounded by the pervasive fear of a Section 166 review – the UK Prudential Regulation Authority’s skilled persons review. The prospect of such an investigation, often perceived as punitive, leads many institutions to over-invest in defensive compliance strategies. Time and resources that could be directed toward robust asset-liability management, strategic forecasting, or customer-focused innovation are instead absorbed by the machinery of regulatory interpretation.”

The cost, he added, is more than financial; its strategic. Institutions become risk-averse not in their lending or investment, but in their thinking. “Innovation slows, judgement is outsourced to consultants, and senior leaders spend more time reviewing spreadsheets than managing real-world outcomes,” he said.

So, a key question arises – how can regulators verify fairness more effectively? The answer, DiRollo believes, may lie in rebalancing the emphasis away from strict rules and towards principles-based regulation.

He said, “Principles-based approaches focus on outcomes rather than methods. They give institutions flexibility in implementation but require justification and evidence that their approach meets the intended goals. This model, while potentially messier to supervise, fosters, substance over form, proportionality and risk-centric oversight. This requires an engaged, expert, and collaborative regulator, but the benefits are profound. It allows institutions to focus on real risk management, not bureaucratic compliance.”

In order to enable this, it is critical for there to be some harmonisation at the data and methodology level. Rather than each bank building its own regulatory model from scratch, industry-wide open standards could offer a shared foundation.

DiRollo said these standards would, amongst other things, define core data structures and taxonomies, provide reference implementations for key calculations and facilitate peer comparisons and sector-wide risk analysis.

He said, “The ultimate goal should be for regulators to provide a clearly defined data taxonomy. Banks would submit granular data in a prescribed format, from which regulators themselves could calculate key supervisory metrics such as RWAs, capital ratios, liquidity outflows, and LCR percentages.”

This model for DiRollo delivers several benefits – such as consistency, as uniform data structures eliminate interpretive divergence, efficiency through allowing banks to focus on ensuring data quality instead of building bespoke calculation engines, and comparability and adaptability.

“By shifting from a model where outputs are submitted to one where outputs are derived by the regulator, the industry can move towards genuine transparency, reduce compliance overheads, and refocus attention on prudent balance sheet management,” he said.

DiRollo concluded, “Verifying fairness in black-box models is not just a technical challenge, it’s a philosophical one. If regulators continue to lead with detailed rules, they will create a compliance theatre that undermines real financial stability.

“A more principles-based approach, combined with shared standards, a common data taxonomy, and transparent supervision, offers a path toward genuine fairness.

Ultimately, the goal must be to restore confidence in financial regulation and supervision – not just in institutions themselves.”

Transparency is key

As AI increasingly powers core financial decisions from verifying identities to flagging fraud regulators face a fundamental challenge, claims RegTech firm AIPrise – can fairness be enforced when the decision-maling engine is a black-box?

The firm said,” These algorithms bring speed and scale, but many operate without transparency. Inputs go in, decisions come out and even the developers may not fully understand what’s happening in between. For regulators, that’s a risk.”

What are the real risks of opacity? The first key area AIPrise finds is regarding bias. “Black-box systems can perpetuate historical inequities, flagging certain businesses or individuals unfairly due to location, demographics, or other sensitive attributes.”

Also bringing issues is regulatory gaps – with compliance frameworks like GDPR and the EU AI Act requiring explainability and traceability, which black-box systems often can’t meet.

Trust is also an issue. “If institutions can’t explain decisions or offer ways to contest them, confidence erodes,” it said.

Blind trust isn’t an option. However, AIPrise said that with the right accountability measures, regulators can oversee black-box models responsibly.

The firm said, “Explainability tools (e.g., SHAP, LIME) can clarify how inputs influence outcomes and third-party audits validate compliance with financial and anti-discrimination laws. Furthermore, human-in-the-loop systems ensure automation does not replace human judgment and outcome monitoring ensures fairness across demographics.”

Fairness in black-box algorithms can still be meaningfully audited even without full access to the model’s inner workings.

This includes for AIPrise, outcome audits, adversarial testing, model probing, regulatory sandboxes and bias and drift monitoring.

As regulators tighten their grip on AI governance, particularly in financial services, the burden of explainability is no longer optional, says the company – its essential.

“This is especially true in high-stakes workflows like KYC and KYB, where opaque decisions can lead to unfair outcomes, missed risks, and loss of trust. At AiPrise, our systems are built with auditable transparency.  Every verification from business identity checks to sanction screenings is logged, traceable, and explainable. That means regulators and clients don’t just see that a case was flagged, they see why it was flagged, what signals were involved, and how the outcome aligns with compliance rules.”

The firm concluded, “We also support human-in-the-loop controls, enabling compliance teams to intervene, review, or override automation when needed, ensuring machine speed never overrides human judgment. That’s why AiPrise is committed to building compliance-first infrastructure.”

Demonstrating fairness and reliability

Regulators can trust black-box algorithms if they are supported by robust testing and validation reports that demonstrate fairness and reliability, claims Arindam Paul, VP of machine learning of Saifr.

He detailed, “Comprehensive test results, including simulations and stress tests, provide evidence of the algorithm’s performance under various conditions. Independent third-party evaluations can also provide impartial assessments. By examining these reports, regulators can build trust in the algorithm’s ability to enforce financial fairness, even without full transparency into its inner workings.”

Arindam explained that regulators can also ensure that these algorithms are subject to regulatory oversight and compliance checks.

He went on, “Additionally, encouraging open dialogue and information-sharing between developers and regulators helps bridge the transparency gap. By incorporating a multifaceted approach, regulators can gain confidence in the algorithms’ fairness and reliability without requiring full transparency.”

Under the hood

For Madhu Nadig, CTO of Flagright, regulators need to see under the hood of any automated decision system.

He explained that by insisting on solid documentation about what data goes in and how the model decides, they can spot bias early. Regular third-party reviews and mock “what-if” scenarios help prove the algorithm treats everyone fairly.

He concluded, “Without those checks, hidden biases slip through, customers lose trust, and firms risk fines. The best audits mix statistical tests for bias, real-world scenario testing and ongoing monitoring to catch any surprises before they reach real people.”

Keep up with all the latest RegTech news here