【正文】 I n t .H i s t o r i c a l L o s s D i s t r i b u t i o n s ( 2 0 0 L a r g e s t B a n k s 1 9 8 4 2 0 0 8 )Validation Example: Alternative Risk Aggregation Models – Distributions of Risk Proxies (Largest Banks As Of 4Q08) Validation Example: Alternative Risk Aggregation Models – Correlations of Risk Proxies (Largest Banks As Of 4Q08) ) 2 0 2x 1 085 0 5x 1 082 0 2x 1 070 2 4x 1 070 2 4x 1 07202x 1 08505x 1 08202x 1 07024x 1 07P a i r w i s e S c a t t e r g r a p h amp。 Jacobs (2021, forthing) ? However, this is putationally demanding, and also requires the elicitation of a prior distribution from an expect, which is very involved ? But if the prior is diffuse, then much uncertainty still remains ? As we don?t see this used in practice currently, we will not further pursue this approach here ? As noted previously, traditional backtesting procedures as applied in market risk VaR models are impractical in an EC model setting ? An alternative approach is to try to assess the accuracy of the EC output by approximating a statistical measure of uncertainty ? ., through resampling or bootstrap methods (Efron et al, 1986) ? But thin data in the tails implies confidence bounds are likely to be wide . EC Model Validation Example: Alternative Risk Aggregation Models ? “Models for Risk Aggregation and Sensitivity Analysis: An Application to Bank Economic Capital”, by Hulusi Inanoglu and Michael Jacobs, OCC amp。 outputs permit evaluation of the relative risk ? Although there is scope improvement, some signs of progress ? Weaknesses of validation particularly when the total capital adequacy amp。 use, weaknesses targeted at evaluation of overall performance might result in banks operating with inappropriately calibrated models ? This could be of concern if assessment of overall capital adequacy is an important application of the model ? Improvements in these areas could include further benchmarking amp。 senior management is necessary for them to understand that there may be greater uncertainty around the output from EC models ? Models not fully validated implies output should generally be treated with extra conservatism ? Understand amp。 technical difficulties ? Clear reporting of such difficulties amp。 algorithms the bank claims to use correctly are understood by staff who develop, maintain, operate and validate the model ? The bank is using in practice the modeling framework that it purports to ? Computer code is correct, efficient and welldocumented ? Data claimed to be used by the bank to obtain its results is in fact being used ? However, this technique is rarely sufficient to validate models, and in practice there is little evidence of it being used by banks for either validation or to explore the degree of accuracy of their models ? Note that replication simply by rerunning a set of algorithms to produce an identical set of results would not be sufficient model validation due diligence Range of Practice in Quantitative Validation: Benchmarking ? Benchmarking the parison of a bank?s EC model to alternative models on the bank?s portfolio ? ., to a vendor model after standardization of parameters ? Among the most monly used forms of quantitative validation used internally ? A limitation of benchmarking is it only provides relative assessments and provides little assurance that any model accurately reflects reality or about the absolute levels of capital ? Therefore, as a validation technique, benchmarking is limited to providing parison of one model against another or one calibration to others, but not testing against ?reality? ? It is therefore difficult to assess the degree of fort provided by such benchmarking methods, as they may only be capable of providing broad parisons confirming that input parameters or model outputs are broadly parable Range of Practice in Quantitative Validation: Benchmarking (continued) ? There may be good reasons why models produce outliers in benchmarking, all of which plicate interpretation of the results: ? May be designed to perform well under differing circumstances ? May be more or less conservatively parameterized ? May differ in their economic foundations ? Comparisons of internal EC are made with varied alternatives: ? Industry survey results ? Rating agency or industrywide models ? Consultancy marketed models ? Academic papers ? Regulatory capital models Range of Practice in Quantitative Validation: Hypothetical Portfolios ? Hypothetical portfolio testing is an examination of either different bank?s EC models on a reference portfolio, or different banks EC output from a given reference model ? This is typically a either a reference model or portfolio external to any one bank ? From a supervisory perspective: permits identification of models that produce outliers amongst a set of banks ? A “model risk management” tool ? Alternatively, this helps supervisors identify banks that are outliers in risk with respect to a reference model ? A “bank portfolio risk management” tool ? In either case this means parison across bank?s models against the same reference portfolio (external to the bank) or of banks themselves (their EC for a given reference model) ? Capable of addressing similar questions to benchmarking, but by different means ? The technique is a powerful one and can be adapted to analyze many of the preferred model properties such as rankordering and relative risk quantification Range of Practice in Quantitative Validation: Backtesting ? Backtesting addresses the question of how well the model forecasts the distribution of outes ? There are many forms of this that entail some degree of parison of outes to forecasts, and there is a wide l