Case Study
Traditionally, testing, validating and monitoring models involves a huge amount of manual work. As well as being slow and hard to scale, that approach also puts pressure on banks’ talent and budgets, detracting from the goals model risk management while increasing its technicalities. We helped a leading US investment bank to decompose the process of validation and to automate key processes for testing and documenting, delivering significant time savings and business benefits
The Challenge
The modelling team at the bank was facing multiple, increasing pressures. On one side, regulators are urging banks to improve the governance and documentation of their models, but on the other, talent within the client’s quant team was at a premium, and there was no appetite to add extra head count. Any solution needed to work across divisions, and it also had to be compatible with each division’s existing templates and processes. It needed to happen without major IT transformation, and it had to comply with the bank’s established approach to managing risk through the three ‘lines of defence’.
Our Solution
First, we separated the process into:
- ‘art tasks’ requiring domain expertise, judgement and communication with business stakeholders, and
- ‘science tasks’ that were more standard and required less judgement
We then focused on optimising and improving the efficiency of the ‘science tasks’, where we proposed decomposing the testing process so that business analysts, programmers and senior quants could focus on subset of tasks in each validation and work on multiple models at the same time. We then proposed an automated solution with three modules focused on testing, documentation and reporting. We focused on identifying small pools of similarities within different divisions. The testing module gave users a simple interface through which they could set tolerance, parameters, data, model and outcome templates, and then run (and, more importantly, re-run) tests with a single click. We set up the system to work with a number of tests as well as model families. To help with documentation, we created and improved a standard template for test results and built a module to automate the importing of test results from C# into Latex and PDF. With the new system in place for the client, everything happens in parallel and hence much more quickly. With automatic updating of only some parts of documents while retaining others, the time gains can be dramatic – particularly compared to working sequentially and manually. Finally, as an additional potential solution, we automated the status of reporting on models, documents and processes. Now, team members could generate reports with one click. Furthermore, we added multiple shared data sources (test code in R/Python/C#) with the output in a range of formats (Word, Latex, SQL, Excel, etc). With this additional feature members of the team could access the same document in a shared document repository. Besides the changes made to any part of documentation, in any data source, were replicated across all formats. The system as a whole featured a simple, intuitive user experience with multi-user access to a shared repository of models and documentation hosted in the cloud. The quant team could now spend more time interacting with model owners and ensuring the goals and principles of MRM were better achieved.
Benefits Achieved
The redesigned approach freed capacity for the validation team and allowed them to dedicate more time to engaging with model users and owners. With improved understanding of model challenges and constraints in the business, the new approach improves MRM where it really matters.
The headline benefits were: shortening the validation cycles of up to 60%, while delivering high-quality, standardized documents.
Automated testing meant that the team could run tests far more quickly. This also allowed them to re-run multiple additional tests at will, with easy customisation each time. With better safeguards against errors and omissions, there were fewer demands on team members’ attention, and operational risk was reduced.
Automating documentation processes removed a huge amount of manual work at a stroke. Team members no longer had to manually enter and update information across many documents, reducing the risk of error. Now they could put their time and energy into tasks that depended on their skill and judgment, such as designing tests and working with validators, model users, sponsors and developers.
Automating reporting made processes for managing model validation smoother and more transparent. Both internal and regulatory reports can now be generated much faster, improving compliance and freeing up time for other tasks.
