Model Governance: New Weaknesses and Potential Remedies

During this year’s Risk Americas event, we had the honor of hosting a panel discussion featuring esteemed panelists from Blackrock, Mizuho, and Columbia University. The conversation centered around the new weaknesses in model governance and potential solutions.

In case you were unable to attend the session live, fear not, we’ve encapsulated the primary discussion points for you below.

The unique challenges of model governance for buyside firms:

The principal model governance challenge facing buyside firms comes from their diversity of investment strategies, clientele and associated models. While there is some regulation on the buyside from the SEC, meeting buyside regulatory model governance expectations poses a few challenges, in contrast to the sellside which requires considerable assistance. Unlike on the sellside, where models are used primarily by lending units, the models developed at buyside firms are utilized by portfolio managers, traders, and a wide array of clients with varying perspectives on world events and product dynamics. Thus, when developing models, teams must provide insights and enable users to interact with the models, reflecting their differing opinions and needs. This need for flexible modeling is a significant challenge, particularly when working with a diverse set of users with different viewpoints.

Adjusting to increasing complexity in the model space:

The increasing complexity in the model space requires accommodating a broader array of opinions and predictions. Traditional models such as standard metric models and regression models are being supplemented with non-traditional elements that consider how different organizations might perform in future regimes. Incorporating unstructured data into analyses, like the rapid refinancing propensity of different mortgage servicers, is an example of this new approach. Another example involves considering potential variations in CLO redemption behaviors based on different types of ownership. All these changes highlight a new modeling paradigm and the challenges that come with it.

The necessity and enablement of automation:

There is a clear necessity for a framework that enables automation, especially considering the increased complexity of current models compared to those from years past. This complexity necessitates more extensive testing at scale. Critical to this effort is having a team proficient in technology while also being capable of understanding model risk and quantitative data. Such a team can provide a functional interpretation of the information, transforming it into actionable insights. In addition, it is crucial to have staff trained on new platforms, enabling efficient validation and quicker model development.

The importance of data governance:

A well-strategized data infrastructure with an organized database is crucial to model performance. Striking a balance between having streamlined data sets and the capability to generate extra, ad hoc work rapidly is critical. Any discrepancies in data consistency or matching with previous records can lead to weeks of unnecessary work. Therefore, proper processes and governance of data, along with an efficient data platform, are vital components of the system.

Skepticism of new model regulation

It is also advisable to develop a healthy skepticism towards evolving model regulations until they’re proven. In the past, there’s often been a significant gap between what’s written down as regulation and what actually gets implemented. It’s easy to claim that we’ll learn from our mistakes, but much harder to move forward in a way that we can legitimately say we have learned from our past mistakes.

Model Information Sharing

There needs to be a bottom-up approach, starting from fundamental analysis. The method to distill this information and communicate it varies across organizations. It’s important not to view it as a single large process involving model development, review, approval/disapproval, and communication. Instead, there should be various check-ins during the process to avoid alarming revelations at the end, such as model disapproval that might lead back to the drawing board.

Examples of past model governance shortcomings

As stated above, there is often a disconnect between what is written in regulation and what is implemented. In one instance a few years back, a securitized product pricing model was generating incorrect results. The issue was traced back to an upstream bond calculator that wasn’t reviewed because it was deemed too simple to be considered a model. This oversight prompted a lengthy debate about whether the calculator qualified as a model or not, and in the end no review was done, thereby highlighting a significant governance failure, and pointing out that guidelines and regulations can sometimes instigate an unproductive approach.

In a second example, a historical value at risk model was being misused. The model inherently included autocorrelation and bias which was overlooked, leading to the model’s results being questioned. Although model risk managers took note and incorporated this into their future analyses, external pressure eventually led to a disregard for this extra step. This homogenization and a “lowest common denominator” approach, which ignores crucial details, has repeatedly proven problematic.

Generative AI in model risk management

There has been a lot of enthusiasm for using advanced AI models, such as ChatGPT, for tasks like producing model validation documents. While these AI models have proven successful in some areas, their application in areas where they haven’t been sufficiently calibrated or tested could lead to significant issues. These large language models (LLMs) have immense potential but should not be expected to completely replace humans. Human expertise will always be required when it comes to complex processes like model risk management.

Conclusion

Managing a multifaceted array of models must be a joint effort, with significant contributions from every element of the model supply chain. However, it’s vital to have clear governance delineating their respective responsibilities. Without this transparency, users and regulators can worry about potential threats to integrity.