Uses and Misuses of Measures for Credit Rating Accuracy
by Alfred Hamerle of the University of Regensburg,
April 28, 2003
Abstract: The New Basel Capital Accord will allow the determination of banks' regulatory capital requirements due to default probabilities which are estimated and forecasted from internal ratings. External ratings from rating agencies play fundamental roles in capital and credit markets. Discriminatory power of internal and external ratings is a key requirement for the soundness of a rating system in general and for the acceptation of a bank's internal rating systems under Basel II. Statistics such as the area under a receiver operating characteristic or the accuracy ratio, are widely used in practice as measures for the performance. This note shows that such measures should only be interpreted with caution. Firstly, the out-comes of the measures depend not only on the discrimination power of the rating system but mainly on the structure of the portfolio under consideration. Thus, the absolute values achieved do not measure the performance of a rating system solely. Secondly, comparisons of the outcomes between different portfolios, different time periods or both may be mis-leading. As a positive result we show that the value achieved by a rating system which predicts all default probabilities correctly can not be beaten.
Keywords: Credit Rating, Basel II, Performance Measurement, CAP, ROC, Accuracy Ratio, Power Curve, Gini.