Modeling continues to play an increasing role in financial services. Indeed, as more and more data becomes available and the technology for using data becomes cheaper and more powerful, more sophisticated quantitative modeling and business analytics are spreading across the industry landscape. As modeling processes come to play a larger role in business decisions, the need for model risk management continues to grow. Financial regulators are increasingly recognizing that need, incorporating model risk management requirements into rulemaking across an increasing variety of financial regulations.

Model errors continue to drive unexpected losses. Some of the more spectacular model risk failures have been well documented. High-profile examples include Long-Term Capital Management in 1998, the evident shortcomings due to using Gaussian copulas in the pricing of structured products in the financial crisis, and the AXA Rosenberg loss.

Bulletin 2000-16 from the US Office of the Comptroller of the Currency (2000), which focused on model validation, was the first formal supervisory guidance on model risk for financial firms. Model risk was subsequently addressed at a high level in the Basel capital rules, developed in greater depth by the Committee of European Banking Supervisors in its guidelines on the implementation, validation and assessment of the advanced measurement and internal ratings-based approaches (Committee of European Banking Supervisors 2006), and expanded in the European Union’s Solvency II rules for insurers (European Parliament and Council of the European Union 2009). Globally, the most important collection of supervisory documents on model risk management (MRM) to date is the US regulators’ consistent, but separately issued, Supervisory Guidance on Model Risk Management (Board of Governors of the Federal Reserve System 2011; Office of the Comptroller of the Currency 2011; Federal Deposit Insurance Corporation 2017: collectively known as the US banking agency guidance).

Model risk has been addressed within a number of other supervisory and regulatory documents. These documents include the Capital Requirements Regulation in the EU (European Parliament and Council of the European Union 2013), the Model Risk Management Guidance of the Federal Housing Finance Agency (FHFA; Federal Housing Finance Agency 2013), and the Securities and Exchange Commission’s (SEC’s) Standards for Covered Clearing Agencies (Securities and Exchange Commission 2016) and Nationally Recognized Statistical Rating Organizations (Securities and Exchange Commission 2014). More recently, the Office of the Superintendent of Financial Institutions in Canada released a guideline titled “Enterprise-wide model risk management for deposit-taking institutions” (Office of the Superintendent of Financial Institutions 2017).

Sound risk management generally requires the allocation of resources according to risk, with riskier areas receiving relatively more. Although the US banking agency guidance does not explicitly refer to risk-sensitive MRM, it does allow that “the range and rigor of validation activities conducted prior to first use of a model should be in line with the potential risk presented by use of the model”. The guidance also states that, “[a]s is generally the case with other risks, materiality is an important consideration in model risk management”. The FHFA’s guidance (Federal Housing Finance Agency 2013) goes into greater detail in this regard, stating expectations for model risk ranking and suggesting areas of activity in which MRM efforts might vary. These include “[t]he level of prioritization, scope, frequency, and documentation of validation activities”. The statements of principle in these supervisory documents are appropriately viewed as endorsing a risk-sensitive approach to MRM. In practice, risk-sensitive MRM is applied by varying the breadth and depth of activities in a number of areas, including model documentation, model validation and ongoing monitoring.

As the importance of model risk and the supervisory requirements around its governance have increased, financial firms have sought to develop means of effectively allocating resources for MRM. Model inventories may be very extensive. Large and complex firms subject to the US banking agency guidance will likely have several hundred or even thousands of models. Given the number of models in many firms and US supervisory expectations for some level of MRM coverage of all models, effective allocation of MRM resources has become an increasingly pressing issue.¹¹ 1 Although the expanded coverage of MRM driven by the US banking agency guidance likely drove the proliferation of model risk-tiering processes across the industry, it had already been enhanced in a number of firms, including Fannie Mae, where one of this paper’s authors developed a risk-tiering scorecard to rationalize the massive model validation effort undertaken to support the financial restatement in 2005–6. For example, a model’s risk level might determine the breadth, depth, priority and frequency of validation activities, such as review of documentation, developmental data and/or performance testing. Similarly, a model’s risk level might impact the extent of model validation testing, such as backtesting, benchmarking, sensitivity testing, stability testing or others. Model risk may also influence the nature of model implementation testing, such as system implementation testing, or the extent of ongoing monitoring activities, such as process verification.

Firms that fail to differentiate models according to risk are unable to implement risk-sensitive MRM. For instance, we have observed an institution with a large model validation unit fail to appropriately implement risk-sensitive model governance for model validation. The validation function found itself overwhelmed by validation exercises conducted entirely to a single high standard. This led to delayed validations that significantly impeded business operations. The model risk unit of this firm also sought to exclude all nonstatistical models from its MRM framework in an effort to address the overwhelming workload. This exclusion led to significant supervisory issues around judgment-based models used for anti-money laundering (AML) and stress testing purposes. This one-size-fits-all approach may also have led to significant numbers of unreported low risk models, subjecting the firm to additional risk and regulatory exposure. Anecdotal evidence suggests that other firms have faced the opposite problem, failing to rigorously validate simple but highly risky models, resulting in severe regulatory issues and reputational events.

In an ideal world, the risk posed by each model would be measured and quantified on a consistent basis. The cost of that risk would be characterized, and risk-mitigating resources would be allocated according to their highest value. In addition to resource allocation, such quantification could help characterize the amount and concentration of model risk within the firm as well as allow transparent reporting of model risk exposure to senior management. The results reported could conceivably be used to create a capital reserve, or buffer, for model risk.

In practice, initiatives to develop approaches for quantifying model risk that can be applied consistently across a model inventory are ongoing and at various stages of maturity. Model risk managers generally use less formal processes – model risk-tiering methods of various types that group models into broad risk groups or tiers but do not further differentiate risk within each tier – to stratify models by risk in order to meet the needs of risk-sensitive model governance frameworks. These model risk-tiering tools implement subjectively derived methodologies that maximize the use of objective data inputs while relying on expert judgment to set at least some input values as well as parameterize the tools. This largely heuristic approach has the advantage of being tractable: the tools can be built and implemented using commonly available data. This has allowed firms to develop and implement simple and transparent approaches to assigning models to broad tiers (that is, to conduct model risk tiering) according to formally applied heuristics.

Few practitioners rank models top to bottom by risk with each model being assigned a unique rank number. Instead, for a variety of reasons – including tractability – model risk ranking at most firms has evolved into a process of assigning models to a small number of risk levels or tiers (ie, three or perhaps four tiers). Risk tiering requires less precision than model-by-model risk ranking, and risk-tier assignments are more stable, changing primarily in response to material changes in risk. In addition, there are not many decisions about risk management that would be improved in a substantial way by a more granular approach to model risk ranking. Current practice is to vary MRM requirements according to risk-tier assignment, with some requirements added as needed for particular models or model-use situations. An appropriately calibrated model risk-tiering tool serves risk managers better than a framework based on a granular risk ranking. Attempts at excessive precision in measuring model risk add complexity but, in practice, bring little benefit.

Processes related to quantitative modeling almost always require at least some discretion on the part of experts, a fact that can be observed through each stage of a model’s life cycle. Similarly, when firms assess relative risks posed by specific models as part of the MRM process, an element of expert judgment is required. Recognizing this fact, management or other expert judgment generally plays at least some role in processes that assign models to risk tiers. Rather than being based entirely on the statistical fitting of models to data, the design of these tools relies upon the qualitative selection of risk drivers and the definition of quantitative boundaries associated with their use.

Although expert judgment necessarily plays an important role in most cases, practitioners also try to use objectively measurable inputs wherever practicable in order to enhance transparency and consistent application. Having the process be transparent within an organization is important, not least because it increases organizational acceptance of the risk-tiering framework. Transparency also reduces uncertainty about model tier assignments, making it easier for model owners and developers to orient their development and documentation efforts toward meeting the standard required by the risk tier to which they expect their models to be assigned. However, some important indicators of model risk, such as model complexity, are exceedingly difficult to measure objectively. Developers of risk-tiering tools must balance the goal of transparency with that of incorporating all significant model risk drivers, since some risk drivers will likely be judgmental.

In order to establish transparency in the development of model risk-tiering tools, developers often solicit input from significant groups of model owners, developers and users, as well as from senior management. In addition to fostering transparency, such an open development process also serves to enhance the organizational legitimacy of the resulting risk tiers as well as to promote consistency in model risk-tier assignments across the organization.

Most firms delegate development and implementation of the risk-tiering methodology to the MRM group, with the head of MRM retaining the authority to override the results of the framework should they appear inappropriate for certain models. Some policies only give the head of MRM the authority to move individual models to a higher (but not a lower) risk tier than that resulting from the risk-tiering tool.

Model risk-tiering tools should be tractable, that is, easily managed and adjusted. Significant factors that affect tractability include the ease or difficulty of obtaining or generating data on the model properties used for risk tiering; the simplicity and flexibility of the risk-tiering process and any internal calculations it uses; and the ease with which the tool can be tuned to bring results more closely in-line with the expert judgment on which the tiering relies. The ability to modify the tool to reflect expert judgment is generally more important than, for example, incorporating complex processes for inputs or rankings in a quest for precision. More tractable tools also tend to be more transparent, so the principles of tractability and transparency reinforce one another.

As in other areas of risk assessment, metrics for model risk can measure inherent risk or residual risk.³³ 3 The Committee of Sponsoring Organizations of the Treadway Commission (2004) states: “Inherent risk is the risk to an entity in the absence of any actions management might take to alter either the risk’s likelihood or impact. Residual risk is the risk that remains after management’s response to the risk. Risk assessment is applied first to inherent risks. Once risk responses have been developed, management then considers residual risk.” In the area of model risk measurement, residual risk might be the focus of a model risk charge, capital charge or haircut. A few firms have implemented a scorecard factor to capture out-of-date or absent model validations or unremediated model issues, both of which are closely related to residual model risk.

However, in practice, most risk-tiering methodologies focus on inherent risk rather than residual risk. Residual model risk takes into account the impact of risk-governance activities. If the primary goal of model risk-tiering tools is to support risk-sensitive model governance, using factors related to the governance of individual models can become circular, with a low level of governance resulting in a model being assigned to a higher risk tier, thereby requiring a higher level of oversight, which could cause a lower risk-tier assignment on the next review (and long-term instability in the tier assignment). In addition, if tier assignment depends on the assessment of residual risk, it will be nearly impossible to compare the risk posed by a reviewed model to an unvalidated model with unknown issues. Tiering that is independent of the results of risk management activities like validation – that is, based on inherent risk rather than residual risk – is better able to treat validated and unvalidated models equally.⁴⁴ 4 This may be particularly important when model risk-tiering systems are first implemented. Many firms first implement a risk-tiering system while working through a model backlog, and the initial model inventory may include many unvalidated models. This creates significant challenges for approaches that include validation results in risk-tier assignments, as the vast majority of models may lack necessary data; Keenan’s (2015) proposed framework, for instance, places critical reliance on model validation and ongoing monitoring results. Inherent risk does not refer to the risk inherent in the model alone. Model risk depends on how a model is used for business decisions. The US banking agency guidance is clear when it defines model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports”.⁵⁵ 5 We note, however, that this definition of use can cause some confusion where firm models are used to provide financial advice to firm clients or made available for public use. While firms may not base their own business decisions on these models, their clients or the general public might. Failure to reasonably control such risk may be viewed by regulators such as the SEC as failures in fiduciary duty and result in financial penalties and reputational harm. Thus, inherent model risk must refer to the risk inherent in a model’s use or uses. In practice, many ranking tools put a higher weight on aspects of a model’s use than on those intrinsic to the model’s design. Any reference to a model’s inherent risk or risk tier should be interpreted as a reference to the risk of the model and its associated uses, and not to the intrinsic riskiness of the model itself.

Measures of risk can be either relative or absolute. Model risk tiering generally reflects an assessment of the risks posed by one model’s use compared with the risks posed by other models in the inventory. The metric produced thus represents relative, not absolute, risk. Moreover, that risk is relative to other models in that specific firm’s inventory, and not relative to the models in use at other firms. This has implications for the definitions of risk-tiering factors and any weights placed on them. For example, an exposure threshold for a regional bank with USD1 billion in assets will almost certainly differ from that used by a systemically important bank with USD1 trillion in assets; if not, then one firm would end up with a much higher proportion of high-risk-tier models than the other. The differences in size and complexity between firms are addressed by differences in the MRM frameworks, which typically dictate more robust MRM activities at each tier for larger and more complex firms than for smaller, simpler ones.

The coverage of the relative ranking process should be the same as that of the firm’s MRM framework. In other words, each model should be ranked relative to all other models subject to the same framework, which usually means the entire enterprise. This facilitates consistent risk-sensitive model governance across the firm. While most firms appear to follow this principle (generally using a single framework, inventory and tiering process across the enterprise), a few may have developed separate risk-tiering tools or calibrations at the business level. This approach could create problems insofar as risk management resources may be misallocated to business units with model inventories that significantly differ from those for which the framework was developed.

For very large, global firms, establishing a firm-wide relative-risk-tiering process may present additional challenges. For instance, a central MRM function at a firm with a very large number of models and businesses may find it difficult or impossible to assess the full inventory well enough to detect material inconsistencies in the rank-ordering results generated by the tiering process. Regulatory challenges may arise for companies operating in multiple jurisdictions. For instance, if a bank’s operations in a foreign country are smaller or less complicated relative to those in its home country, the distribution across risk tiers of models used in the foreign country may be perceived by a foreign regulator as excessively skewed toward the lower risk tiers. Such complications, while tractable, are beyond the scope of this paper.

While the boundaries between risk tiers, and thus the distribution of models across risk tiers, may emerge naturally from observations of model profiles, the initial application of a model-tiering tool may produce an unreasonable distribution of models across risk tiers.⁶⁶ 6 The terms “model profile” and “model profile elements” refer to data about a model and exclude data used in the development or operation of the model. In practice, a commonly encountered issue is an excessive concentration of models in the highest risk tier. While this is not necessarily an incorrect outcome, such concentrations raise natural questions about the design and calibration of the framework. Since the tool is meant to produce a relative risk tiering of models to support risk-sensitive model governance, such clustering may suggest weaknesses in the tool’s ability to discriminate between models presenting different levels of inherent risk. In that case, the tool may not be fulfilling its purpose and could require further refinement.

For a number of reasons, the distribution of models across risk tiers will vary widely between firms. Commonly observed practices within the banking industry suggest a few guiding rules of thumb. In a typical system with three or four tiers, any risk tier containing fewer than about 10% of the total number of models in the inventory is probably too narrowly defined to be useful. Similarly, for any of the higher risk tiers, any tier containing more than about 30–40% of the total number of models is likely too broadly defined to provide the degree of risk differentiation needed for effective MRM. However, since the distribution of models across tiers is set arbitrarily by the construction of the tool, tool developers must consider the resulting distribution of models across risk tiers as part of the tool’s calibration.

Firms look for model risk tiering that is both stable and predictable. Given the extent to which risk-tier assignments may drive model development and validation activities, model owners and developers will frequently request a determination of a proposed model’s risk-tier assignment before or during the development process so that development activities can be tailored to meet the relevant expectations. With an established process, indicative risk tiers can be quite easy to produce. Challenges may arise if the model profile changes once the model is made available for use, particularly if users embrace a model more or less enthusiastically than expected, or if the volume of business at which the model is directed changes substantially within a short time. More problematic predictability issues may stem from disagreements or miscommunication between the model owner and the MRM team regarding judgmental model profile elements needed by the tiering tool. For less established processes, changes made to the tiering tool during model development may drive changes in models’ ultimate risk-tier assignments.

For models that are in use, instability in risk tiers can be even more problematic. Reassigning a model to a higher risk tier may require significant incremental work on the part of model owners, developers and risk managers, while reassigning a model to a lower risk tier may be perceived as reducing the value of similar work already completed. A process that results in frequent revisions to tier assignments after model implementation can significantly undercut internal acceptance of the entire model governance framework.

A desire for stability in tier assignments does not override all other considerations. The risk tiers assigned to models can, and in some cases should, change over time if new information about model risk comes to light, or if a firm identifies a need to revise the MRM framework. The assignment of a model to a different risk tier can be caused by changes to thresholds or other aspects of the model-tiering tool, or by changes in the model profile elements utilized by the tool. This latter change may be caused by the discovery of additional uses, the correction of data errors or the reevaluation resulting in changes to subjective elements of the model profile.

Firms generally find that the stability and predictability of model risk tiering increases over time as staff gain more experience with a given tool and its inputs, and as awareness of the tool spreads within the firm. The key to establishing a stable and predictable model risk-tiering tool appears to be a methodical development process that provides maximum transparency regarding both the tool’s methodology and its results to all stakeholders as development progresses.

A few firms have elected to implement a model risk tree (MRT) approach like that described in Mankotia and Joshi (2013), illustrated in Figure 2.

Simple MRTs such as this one have the advantage of being easy to understand and transparent. In addition, since they do not produce a granular metric, MRTs are unlikely to convey false precision.

This MRT evaluates the model-use dimension with a two-step process before using a notional exposure threshold to make the final assignment. The use dimension first evaluates whether the model measures risk, price or value – an indicator of the materiality of the model’s domain coverage – with a “no” answer automatically resulting in assignment to a low risk tier, while a “yes” moves the evaluation to the next stage. The second stage of the use dimension evaluates whether the model is used in critical decision making or for regulatory or financial reporting. If so, the model is assigned to the high or moderate risk tier, depending on notional exposure addressed by the model relative to a threshold; if not, the model is assigned to the low or moderate risk tier, again depending on notional exposure relative to the threshold. Later boxes rely on judgment regarding the model’s use and exposure level.

As may be evident from the description, this particular decision tree implementation relies heavily on judgmental inputs. It also relies on binary categorical outcomes and therefore does not allow for more than two answers to any question. This is a common approach to decision tree construction both because multiple categorical outcomes (also known as multiple-level categorical variables) can make decision trees very complicated very quickly – and thus difficult to represent and interpret – and because decision trees may become biased with the inclusion of categorical variables with multiple levels (see Deng et al 2011). Visual representation of decision trees can get even more challenging when multiple-level categorical variables and larger numbers of variables are combined.

In practice, in part due to these challenges, designers tend to keep decision trees simpler than scorecards, even though very complex decision tree models have been developed in various disciplines (see, for example, Lewis 2000). While simple is good, oversimplification can create problems for a tiering process. For instance, in the uppermost diamond of Figure 2, the use of a binary variable identifying whether the model measures risk, price or value may create a very strong binary effect, where a more continuous multiple-level categorical variable would give a more nuanced result that aligns better with expert judgment. In practice, such binary materiality variables can create tools that inadequately differentiate model materiality and result in a concentration of model assignments in higher tiers (clustering).

Figure 3 shows a sample model risk scorecard (MRS). This scorecard uses a factor/component approach with four factors. Some factors are represented by single components while others utilize multiple components. The “value range” column gives potential values for the model inputs, some of which may be directly sourced from the model profile while others may require preprocessing to convert variables with continuous or widely varying values into more limited categorical variables. Examples of variables that must be converted to categorical types include estimated dollar exposure, number of users and number of input variables. The “weight” column gives the percentage weight of each factor (in yellow) in the final score, along with the weight of each component of the factor value (in pink). In addition to the four factors, this MRS includes two adjustments: a “capital stress testing” indicator, which can push a model into tier 1 regardless of its score on other factors, and an MRM adjustment, a score override allowing the head of the MRM group to add up to 50% of the total possible value to the base metric. The MRS produces a final risk score to drive the risk-tier assignment, as seen at the bottom of Figure 3. The process of setting preprocessing thresholds, component and factor weights, and total risk-score tier-assignment thresholds is addressed below in our discussion of calibration.

The sample MRS in Figure 3 is clearly more complex than the sample MRT in Figure 2. It mixes binary variables with multiple-level categorical variables. It also incorporates a weighting scheme that may be less intuitive than that implicitly incorporated into the decision tree structure. However, the MRS has other advantages. For firms that wish to include several categorical variables with multiple levels, the MRS will likely prove easier to represent visually, and thus easier for users to understand.

To understand how this scorecard works, consider the calculation of the “operational factor”. This factor has a weight of 10% in the final score, with 50% contributed by the “EUC implementation” component, a binary variable with potential values of 0 or 1, and the other 50% contributed by the “number of users” multiple-level categorical variable with potential values of 1, 2, 3 or 4. The total value of this component in the final score for model 1 is $(1\times 50\%+2/4\times 50\%)\times 10\%=7.5\%$. Since the total value before adjustments is 100, the “operational factor” contributes 7.5 points to the total score, as can be seen in the “operational factor” line for model 1 in the “model scores” box at the bottom of Figure 3.

The final “model tier” is recorded in the highlighted boxes in the bottom line of Figure 3. For this scorecard, the thresholds are 50 points for tier 2 and 80 points for tier 1. Two interesting cases are models 2 and 3. Model 2’s total score of 59 out of 100 points would place it in tier 2, but the model has been assigned to tier 1 because the “capital stress testing” override is set to 1, which automatically assigns all models used in capital stress testing to the highest tier. For model 3, the total score is $22.5+7.5+3=33$ points. In this case, MRM has decided to add an additional 10 points to bring the score up to 43, but this does not affect the final model tier assignment, since the total score remains below the 50-point threshold.

For ease of communication, we will use the nomenclature from Figure 3, calling the general dimensions of model risk that firms wish to incorporate into their risk-tiering tool “factors” and the specific variables chosen to indicate those risk dimensions “components”.

Decisions regarding which components to include are usually best made in conversations or workshops led by the MRM group and involving a wide range of stakeholders. This wide participation not only encourages acceptance and ownership of the process across the organization but also gives valuable information regarding the difficulty of obtaining the different data elements and assessing their reliability and meaningfulness as risk indicators. It is good practice to begin the collection process by targeting multiple components for each factor, since the data-collection process is unlikely to result in a full set of data for every component targeted for use in the model-tiering tool.

Once the proposed list of components is settled, primary responsibility for data collection and maintenance generally falls to model owners (and their staff) regardless of whether model owners are developers or users. Of course, the model owner may not be the ultimate source of that data, which likely comes from a variety of sources: in many cases, from model users. The MRM group may create a questionnaire in order to solicit the desired model profile elements from model owners and others. Respondents to the questionnaire are generally required to certify their information on a “best efforts” basis and revise and recertify it annually or in the event that the model is revised or its uses changed. The MRM group should carefully review responses for completeness and reasonableness. The tiering-tool developer should maintain a dialogue with model owners to discuss missing or questionable values and to ensure that data is in a usable form and comparable across models.

While much of the model profile data used in tiering tools is easily observable, many practitioners seek to add one or more variables that are judgmental. For instance, some model-tiering tools require an estimate of the impact or contribution of each model on a larger calculation, such as regulatory capital. While intuitively appealing, such variables can be problematic if the level of actual or potential impact of a model on its outputs is unconstrained. For example, when experts attempt to determine the maximum impact of catastrophic model errors in such situations, they frequently find that number to be very large, or very hard to quantify, or both. Such exercises are often reliant on expert heuristics. Disagreements regarding such heuristic inputs can be very challenging to resolve. In part because of these challenges, many tiering-tool development processes revert to more easily observable metrics, but this sometimes comes at the expense of useful information.

Different types of components require different treatment. Quantitative variables with a wide range of potential values are generally converted into categorical variables represented by a number of integer values, such as categories one through five. Many judgmental variables are also represented in practice by an integer scale; in that case, narrative descriptions of the values on the scale may need to be developed in discussions with stakeholders. Other components may be reduced to simple binary measures. Regardless of which form a component takes in the tool, it is best to collect the input data in the most granular form possible, to maximize its information content and the flexibility of the tool-construction process.

Once the input data is collected, the usual data-hygiene processes must be completed, such as cleaning data and converting text entries to categorical variables. In practice, this stage can be quite time-consuming, as respondents are likely to interpret profile-element requests differently, regardless of the care taken in the development of questionnaires or other instruments, and the descriptive information provided in response may take a wide variety of styles. Enough time should be allowed for significant back-and-forth between the tool developer and respondents in order to develop a data set that is as robust and consistent as it is practical.

Once the input data has been collected, the structure of the model-tiering tool must be chosen. One of the most fundamental decisions is between the tree and scorecard approaches discussed above: that is, between using an MRT and using an MRS.

For an MRT, the design challenges related to tool structure can be much more significant than those for an MRS. For trees, decisions such as in which order to place nodes, which nodes belong on which branches, and which branches to prune are inseparable from calibration decisions such as where to set thresholds and the assignment of final branches to risk tiers.

For an MRS approach, developers must choose whether to weight and add up risk-component variables separately or to add them up by factor subgroups. Additional choices include whether or not to include automatic overrides, whether to include actual values for continuous variables (such as dollar volume) or convert them to a more constrained form (such as categories or ranges), and whether to limit the maximum potential contribution of variables.

In addition to choosing between the MRT and MRS frameworks, tool builders also need to decide whether to make tier assignments at the model level or at the model/use pair level. Tiering models at the model/use pair level means treating the combination of a model and each of its business uses as if they were unique models, and tiering them on a stand-alone basis. As a result, a single model may receive multiple risk-tier assignments depending on use. Firms using this approach often base MRM requirements on the highest tier assigned to any model/use pair. More commonly, however, firms apply their tiering tools to create a single tier for each model, taking into account all of a model’s uses at once. Models with more model uses are more likely to be assigned to higher risk tiers.

Which approach is more appropriate depends on the MRM framework and practices of the individual firm. Since the depth and frequency of validation activities are frequently driven by a model’s risk-tier assignment, firms that conduct separate validations for different model uses might reasonably choose to utilize the model/use pair approach, while those that conduct validations covering all of a model’s uses might use the model-centric approach.

Many firms apply a bucketing scheme to each component variable with a wide or continuous range of potential values, converting it to a categorical variable in order to reduce data dimensionality. This may be done simply to ease the tool-construction process, to constrain the impact of the variable on the final value or to accommodate a variable believed to bear a nonlinear relationship to model risk when expressed in its natural units. For instance, a variable such as the dollar value of exposures covered by a model might be seen as having a greater risk-differentiation power for each dollar change at low values than it does at very high values. Alternative approaches to such nonlinearity, such as converting data using a fractional exponent to convey the change in marginal impact as values rise, are not generally employed.

If particular data items are not available for some models, the developer must decide whether to drop the variable from the tool or to set an arbitrary value for the initial calibration process. Through this iterative process, the risk-tiering tool may be further revised.

Choosing a set of inputs to represent the components for the various risk factors does not determine how models will be tiered. The same set of factors and components can lead to an almost endless variety of tier assignments depending on how the variables are combined and weighted. The determination of an appropriately weighted combination is the essence of calibration. In this section, we address the calibration of MRS-style model risk-tiering tools.⁹⁹ 9 The calibration of MRTs, while analogous, will differ in important aspects.

Depending on the structure of the MRS, calibration may be required at multiple levels:

• •

  the thresholds of any buckets for individual variables;

• •

  the weighting of components;

• •

  the weighting of factors; and

• •

  the setting of thresholds for converting the final score to a recommended risk-tier assignment

Once the approach to the MRS is chosen, developers commonly use one of two approaches to develop their initial model calibration: expert judgment or regression analysis. Under the expert-judgment approach, the developer sets parameters that are intuitively appealing and then experiments with varied parameters to reach the desired results. Developers will generally look for results to meet the following constraints.

• •

  Risk-tier assignments generally align with any existing assignments from a legacy approach or conform to intuition.

• •

  Each component has a meaningful impact on the result (eg, no component has a weight so low that it would never be expected to determine a risk-tiering result).

• •

  The parameters and structure of the MRS are intuitive.

In firms without a preexisting rated-model inventory, the MRM group usually applies considerable judgment regarding calibrations, as shown on the left-hand side of Figure 4. Legitimacy for calibrations in this environment is generally established by communicating the results of the calibration exercise to stakeholders, inviting feedback and emphasizing the overall reasonableness of the risk-tier assignments rather than extensively negotiating the details of individual component or factor weights.

In firms without preexisting model risk-tier assignments, the first round of such exercises frequently results in an excessive number of models being assigned to the highest risk tier. Unless the model universe is such that very few models are utilized in a large number of applications, such a distribution should not be accepted. As discussed above, the MRS is a tool for distinguishing the riskiness of models within an inventory; the placement of too many models into a single risk tier suggests that it is not differentiating risk as intended. In that case, further calibration or changes to the tool’s structure are likely necessary before the tool can be considered finalized.

For firms that have a legacy approach and therefore begin with an existing distribution of models across risk tiers, the calibration of a new framework may proceed somewhat differently, as illustrated by the flow on the right-hand side of Figure 4. In such cases, maintaining the preexisting relative ranking of models is often seen as a desirable feature for the new system, and risk managers may aim to minimize the number of model risk assignments that change under the new tool. Assuming that some degree of confidence in current tier assignments is justified, controlling changes in risk tiers has the advantage not only of minimizing the incremental work to implement the new approach but also of incorporating and making more systematic the information contained in preexisting processes, while increasing the general organizational acceptance of the new tiering methodology.

Calibration to replicate an existing distribution is usually accomplished through trial and error, using either expert judgment or alternative functional forms to bring the distribution of models across risk tiers close to that of the current inventory. In some cases, firms can use regression-based approaches to determine a combination of model profile elements and weightings that produces an appropriate risk tiering of models. If the existing model distribution across tiers is regarded as a valid starting point, then a well-tuned statistical and/or iterative calibration process will replicate the current model risk-tier assignments as closely as possible using the model profile elements collected. This process is complete once the resulting model risk tiers and calibration parameters are deemed reasonable by management.

Although the initial tier assignments serve as a valuable anchor for a new calibration, the old and new tier assignments should not be required to be, or expected to be, identical. The consistency imposed on model risk assessment by a well-structured MRS should be expected to result in some movement of models between model tier assignments. Considerations that may reasonably affect the distribution of models across tiers commonly include the following:

• •

  the extent to which individual models are reused for different products or in different applications (eg, a firm in which a small number of models are utilized broadly will have a smaller inventory, with a larger proportion of high-risk models, than one in which each use tends to generate its own model);

• •

  the comprehensiveness of the inventory, since models omitted from the inventory are likely considered less crucial and would thus likely fill out the lower risk tiers;

• •

  the alignment of tier-specific MRM requirements with the risks posed by the models assigned to each tier; and

• •

  the proportion of the inventory that must be assigned to the top risk tier due to supervisory pressure (eg, examiners may suggest that all models used in stress testing should be assigned to tier 1).

Once a near-final calibration is established, the MRM group should lead a process to resolve any issue with new risk-tier assignments through discussions with model owners and business leads. At the end of this process, some firms will also choose to set overrides in the MRS to guarantee a minimum tier assignment for models of certain types or with certain uses, such as those required in regulatory-compliance activities.

Our experience has been that initial model calibration frequently results in risk-tier assignments for the model inventory with which managers are dissatisfied, particularly at firms with preexisting model risk-tier assignments. While a tiering tool should not be required to perfectly replicate an inventory’s preexisting tiering, it is important that the tool be able to generate tiers that are generally consistent with expert opinion where that opinion is itself internally consistent. Depending on the choice of inputs, calibrating an MRS to be broadly consistent with current risk-tier assignments may prove easier than doing so with an MRT.