Are Your Machine Learning Models Wrong?

Beware the Double Shock Waves Hitting the UK & EU Financial Services Sector

In addition to the very real negative impact on every person around the world, the COVID-19 pandemic is driving business disruptions and closures at an unprecedented scale. Enormous government stimulus programs are resulting in explosions in fiscal deficits, regulators are relaxing capital constraints on banks and central banks are supporting economic stability with a range of interest rate cuts and other stimulus measures.  

While all industries are impacted, the retail, travel, and small business sectors are probably the worst hit. The key near term challenge for the financial services sector will be the surge in credit losses that could persist into 2023. Fortunately, most of the banking system seems to be adequately capitalized to absorb this spike in credit losses – though there will be some institutions that will have to address likely capital shortfalls.

As an example, McKinsey estimates the US credit card losses under two economic scenarios:

   A1) Virus Contained: This has GDP recovery to the pre-crisis level by year-end 2020.

   A3) Muted Recovery: This has GDP fully recovering by the end of Q1-2023. 

They estimate that losses could easily triple under the Muted Recovery scenario.

Credit Card Losses Forecast

The UK and EU’s Double Shock Waves

The First Shock Wave – COVID-19:  COVID-19 is a global pandemic that has halted most of the world economies. It has disrupted nearly every aspect of our lives both personal and professional. The contagion effects of COVID-19 are both medical and economic in nature and at a scale not seen since the Spanish Flu from 1918 to 1920.  The duration of this pandemic is yet to be known as are the likely future status of society and the global economy.  What the ‘new normal’ will be once the pandemic has been controlled is a key topic of discussion and analysis.

The Second Shock Wave – Brexit:  For the UK and the European Union, 2021 will most likely result in a second wave of disruption and uncertainty.  This will be driven by Brexit, the UK’s departure from the European Union.  This second wave of disruption could not come at a worse time for both entities which will probably be just starting to recover from the COVID-19 crisis

In isolation, Brexit would be a momentously disruptive event, but when this is conjoined with the COVID-19 crisis we have a classic double shock wave.   This added complexity at the very least probably extends the high level of economic uncertainty that the UK and the EU must manage while the rest of the world will be more singularly focused on their recovery from the COVID-19 pandemic. 

In these circumstances, concerns arise about the accuracy of Machine Learning (ML) models, how recovery will be different for the UK and EU relative to the rest of the world, and what financial institutions should do to address these concerns.

Are ML Models Wrong?

A slightly provocative question yet critically important in the current situation of unprecedented uncertainty!  Many of the ML models in production today were estimated using large volumes and deep histories of granular data.  It will take some time for existing models to be re-estimated to adjust to the new reality, as well as to the unknown situation of a ‘new normal’.  The most recent example of such complications and abnormalities at a global scale was the impact on risk and forecasting models during the 2008 financial crisis.

A key consideration in the current situation is whether this is a ‘Structural Change’ or a once in a hundred years ‘Tail Risk Event’.  If the COVID-19 pandemic is considered a one-off ‘tail risk’ event, then when the world recovers, the global economy, the markets, and businesses will operate in a similar environment to the pre-COVID-19 crisis.  The challenge in this case is to avoid models from being biased due to the once-in-a-lifetime COVID-19 event.  

On the other hand, a ‘structural change’ represents the situation where the pandemic abates, and the world settles into a ‘new normal’ environment that is fundamentally different from the pre-COVID-19 world.  This ‘new normal’ environment requires institutions to develop entirely new ML models utilizing expanded or alternative data that provides sufficient data to capture this new and evolving environment.   

What Should Financial Institutions Do in this Situation?

Financial Institutions have a range of options they can explore to address these challenges.  I will touch on a few basics but also want to focus on the opportunity for institutions to take a longer-term view in implementing a strategy that enables a set of expanded capabilities to help prepare for the next crisis.

1.Modifying existing models: This will be the starting point for all data science teams and can range from using the latest data elements to modify current models while also creating scenario-based projections adjusted for various levels of model bias.  There are a range of techniques that can be utilized including a Bayesian approach to capture expert judgement into the models.

2.Stress Testing: Stress testing provides institutions the opportunity to gain a clearer understanding of their hidden vulnerabilities. This is particularly true with respect to unknown model inaccuracies.  Chief Risk Officers should think strategically about both embedding stress testing across all lines of business and increasing the rigor of these from a few static scenarios to an automated system running hundreds of dynamic stress testing scenarios.  Extending this to a prescriptive analytics framework will most like be a new enhancement in the next generation of stress testing platforms.

3.Industrialization of ML: This is the ideal time to invest in building out a platform that supports the entire ML lifecycle in order to enable the industrialization of ML.  This goes beyond the model development workflow and incorporates a full enterprise-wide ML Operations layer.  Many institutions are already developing these types of capabilities. 

Figure 2 is a high-level workflow illustration of the  Cloudera Machine Learning  – CML.  This is available on the industry’s first Enterprise Data Cloud platform called  Cloudera Data Platform (CDP). CDP provides consistent model management, security, and governance across hybrid, multi-cloud and on-premise supported environments enabling our customers to have a holistic enterprise view of their ML model landscape.

This type of investment in technology capabilities and expert staff will yield dividends, during these tumultuous times and in accelerating the adoption of ML-driven solutions to support all aspects of an institution’s business. This type of environment will enable financial services institutions to build, validate, productionize, manage and monitor all their models across the entire enterprise.

ML Workflow

4.Prescriptive Analytics: A complementary approach to existing ML models involves adopting prescriptive analytics driven by simulation capabilities into the decision-making process.  One new simulation approach that is starting to be adopted in the financial services industry – especially during the current crisis – is Agent-Based Modelling (ABM).  An ABM model is a bottom-up simulation approach to the modeling of complex and adaptive systems with heterogeneous agents. An agent can be a person, product, institution, marketplace, or the economy.

While agent behavior and decision points are calibrated against historical data, the ABM approach can support projecting thousands of future scenarios that are not dependent upon historical data limitations – as with ML. This allows decision-makers to evaluate the impact of various shocks, feedback effects, alternative business strategies, and regulatory changes.  This holistic view provides decision-makers with a more robust view of potential future outcomes and the factors driving each outcome.  This is also a highly effective approach toward creating a largely automated dynamic model stress testing environment.

In the current environment, ABMs are complementary, critical tools to help businesses evaluate how best to assess the optimal strategies to pursue given a wide range of future “new normal” outcomes.   This is even more critical for the UK and EU financial services sector given the conjoined Double Shock Waves of COVID-19 and Brexit.

[1] Higginson, Matt and Frédéric Jacques, Nick Malik, Rohit Singh. “What next for US credit-card debt?”  McKinsey Article, May 23, 2020. (

Richard Harmon
More by this author

Leave a comment

Your email address will not be published. Links are not permitted in comments.