【正文】 iterature on the subject ? However, weak power of backtesting tests for models of risk that quantify high quantiles has been noted ? ., for portfolio credit models see BCBS (1999) ? Variations to the basic backtesting approach which can increase the power of the tests have been suggested in the literature: ? Backtesting more frequently over shorter holding periods (., in market risk using a oneday standard versus the 10day regulatory capital standard ? Using crosssectional data on a range of reference portfolios ? Using information in forecasts of the full distribution ? Testing expected values of distributions as opposed to high quantiles Range of Practice in Quantitative Validation: Backtesting (continued) ? Backtesting is useful principally for models whose outputs are a quantifiable metric with which to pare an oute ? However, some risk measurement systems in use have outputs cannot be interpreted in this way and cannot be backtested ? Such risk measurement approaches not amenable to outesbased validation might nevertheless be valuable tools for banks ? ., rating systems, sensitivity tests and aggregated stress losses. ? The role of backtesting for such models, if used, would need elaboration ? In practice, backtesting is not yet a key ponent of banks39。 sensitivity testing to described later ? However, that checking of model inputs is unlikely to be fully satisfactory since, every model is based on underlying assumptions ? The more sophisticated the model, the more susceptible to model error, so checking input parameters will not help here ? However, model accuracy and appropriateness can be assessed, at least to some degree, using the processes described in this section Range of Practice in Quantitative Validation: Model Replication ? Model replication is useful technique that attempts to replicate EC model results obtained by the bank ? This could use independently developed algorithms or data sources, but in practice replication might leverage a bank?s existing processes ? ., run a model of the same type or class on a the bank39。 and cannot be altered without fundamentally changing the model ? To illustrate, these assumptions could be about: ? Fixed model parameters (PDs, correlations or recovery rates) ? Distributional assumptions (margins, copulae amp。 outputs are interpreted ? This should take account of the specific implementation framework adopted and the assumptions underlying the model and its parameterization ? Data quality checks refer to the processes designed to provide assurance of the pleteness, accuracy and appropriateness of data used to develop, validate and operate the model ? ., Review of: data collection and storage, data cleaning of errors, extent of proxy data, processes that need to be followed to convert raw data into suitable model inputs, and verification of transaction data such as exposure levels ? While not traditionally viewed by the industry as a form of validation, increasingly forming a major part of supervisory thinking Range of Practice in Qualitative Approaches to Validation (concluded) ? As all models rest on premises of various kinds, varying in the degree to which obvious, we have examination of assumptions ? Certain aspects of an EC model are 39。 how these vary over time ? Some EC models are of risk aggregation models where estimates for individual categories are bined to generate a single risk figure – There may be no best or unique way to do this aggregation ? Since many of these assumptions may be untestable, it may be impossible to be certain that a model is conceptually sound ? While the conceptual underpinnings may appear coherent and plausible, they may in practice be no more than untested hypotheses ? Opinions may reasonably differ about the strength or weakness of any particular process in respect of any given property Validation of EC Models: Introduction to Range of Practice ? While we will describe the types of validation processes that are in use or could be used, note that the list is not prehensive ? We do not suggest that all techniques should or could be used by banks ? We wish to demonstrate that there is a wide range of techniques potentially covered by our broad definition of validation ? This is creating a layered approach, the more (fewer) of which that can be provided, the more (less) fort that validation is able to provide evidence for or against the performance of the model ? Each validation process provides evidence for (or against) only some of the desirable properties of a model ? The list presented below moves from the more qualitative to the more quantitative validation processes, and the extent of use is briefly discussed Validation of EC Models: Range of Practice in Qualitative Approaches ? The philosophy of the use test as incorporated into the Basel II framework: if a bank is actually using its risk measurement systems for internal purposes, then we can place more reliance on it ? Applying the use test successfully will entail gaining a careful understanding of which model properties are being used and which are not ? Banks tend to subject their models to some form of qualitative review process, which could entail: ? Review of documentation or development work ? Dialogue with model developers or model managers ? Review and derivation of any formulae or algorithms ? Comparison to other firms or with publicly available information ? Qualitative review is best able to answer questions such as: ? Does the model work in theory? ? Does the model incorporate the right risk drivers? ? Is any theory underpinning it conceptually wellfounded? ? Is the mathematics of the model right? Range of Practice in Qualitative Approaches to Validation (continued) ? Extensive systems implementation testing is standard for