Document View

Since their commercial introduction in the early 1990s, residential property-valuation models have significantly altered collateral-valuation methods in mortgage origination and underwriting. Despite property-valuation models' increased adoption and in spite of ongoing opposition from the more traditional elements of the appraiser community, lenders still face major challenges in convincing investors and regulators of the value of valuation models. From the start, many vendors promoted property-valuation models as outright replacements for traditional appraisal reports. Property-valuation models depend in whole or in part on real estate ownership and transfer data compiled from public record sources. In the early 1990s, property-valuation models were positioned as tools capable of replacing human appraisers for a significant percentage of mortgage applications. The single most important drawback of property-valuation models is their inability to gauge the property's condition or to take into consideration buyer-seller motivation. There is no question that property-valuation modeling has permanently changed collateral-assessment practices for a large segment of mortgage lending.

SINCE THEIR COMMERCIAL INTRODUCTION IN THE EARLY 19905, RESIDENTIAL property-valuation models have significantly altered collateral-valuation methods in mortgage origination and underwriting. However, as early participants in the development and deployment of valuation models, we see profound weaknesses in the approaches taken by lenders, vendors and regulators to understand and apply this new technology. This is particularly true in the areas of testing and implementation. * In our view, the conventional process by which automated valuation model (AVM) products are evaluated and compared is fundamentally flawed. Much emphasis is placed on the esoteric aspects of property-valuation modeling and third-decimal-point precision in comparing model performance, yet underlying testing methodologies produce results that are neither representative nor predictive of future performance. * Also, the application of property-valuation models to collateral-assessment processes has been disappointingly rudimentary, in our view. In a sense, the industry has struggled to replace a complex, multistage appraisal process with a single mathematical calculation-with predictably disappointing results. * At the root of the industry's struggle with property-valuation models is a lack of understanding of probability theory and a concomitant desire to seek validation of the accuracy of every individual estimate. If we applied that same philosophy to rating the likelihood that a given airline flight might not reach its destination, attendance at the upcoming Mortgage Bankers Association (MBA) Annual Conference would be disappointingly sparse.

Enlarge 200% Enlarge 400%

Although estimates vary, by some accounts as much as 50 percent of originations may involve the use of valuation models as an instrument of loan underwriting. Much of this use is concentrated in the refinance and home-equity lending sector, with purchase-mortgage lending generally relying upon conventional appraisal practices.

Despite property-valuation models' increased adoption and in spite of ongoing opposition from the more traditional elements of the appraiser community, lenders still face major challenges in convincing investors and regulators of the value of valuation models.

While many stakeholders have expressed concern about the performance and validity of property-valuation models, it remains to be seen whether these concerns are valid. We argue in this article that if the technology is to be advanced, the focus should be directed at how property-valuation models are validated and deployed rather than on the arcane aspects of property-valuation modeling and testing.

In our view, two areas warrant further discussion: model testing and validation, and redesign of the collateral-valuation workflow process.

New ground-or not

From the start, many vendors promoted property-valuation models as outright replacements for traditional appraisal reports. Lenders were eager to adopt these models to close loans faster and cheaper against a backdrop of high demand and market competition. Yet increased usage has given rise to concerns by the investors and the regulatory agencies about the potential impact on the quality of loans underwritten using valuation models.

Arguably, this is not new ground. Validation of statistical models was addressed in the previous decade, when mortgage origination transitioned from heads-down, industrial underwriting to the use of credit-scoring models when examining consumer credit. However, testing and validation of credit scores fell under the domain of risk management, while responsibility for evaluating and implementing property-valuation models has come to rest with the appraisal departments. It's fair to say they may not be ideally positioned for the task, given the traditional emphasis of these departments on the accuracy of every single property appraisal.

This is not to say that appraisal departments should not influence validation and deployment of property-valuation models. On the contrary, to increase business efficiency while retaining integrity, the best elements of the traditional appraisal process should be combined with the strengths of property-valuation models. With that said, it is neither appropriate nor necessary to define a new paradigm for model validation to address the concerns of investors and regulatory agencies.

Model testing and validation

Property-valuation models depend in whole or in part on real estate ownership and transfer data compiled from public record sources. (Tax assessment offices provide property location and ownership, property type and, in some areas, physical property details such as square footage and room counts. Recorders' and registries of deeds are the source of property transfer and financing details.) Some models maintained by private investors and the government-sponsored enterprises (GSEs) incorporate a significant (and usually exclusive) source of appraisal data. These private data sources extend both the reach and the performance attributes of valuation models.

Unlike the highly regulated data repositories that provide the source data for credit scoring, data for property-valuation modeling is fairly accessible. It is quite feasible to establish a property-valuation modeling business with initial capital outlays that do not exceed $3 million to $4 million. This investment includes the cost of licensing a national data set as well as the cost of technology and personnel. This accessibility lowers barriers to entry and raises the level of competition in the property-valuation space. From a pricing perspective, this competition has advantaged users; however, it has arguably stunted the growth of what otherwise would be a more sophisticated and mature industry.

This lack of sophistication is nowhere more apparent than in the current business practices involving model testing and validation, and the quest for uniform industry standards. Again, we stress that it is not our view that property-valuation models should not be scrutinized more carefully. If anything, over the course of the last 10 years users have generally raised their awareness of where and how these models work and, more important, where they don't work. However, to differentiate their offerings, most vendors have stressed complex and esoteric aspects of modeling techniques and product features rather than emphasizing data quality and content-elements that have a far greater effect on model performance.

Enlarge 200% Enlarge 400%

Among users, model validation and testing are not sufficiently supported by staff trained in quantitative or modeling techniques, including the rudiments of sampling. Testing is, to some extent, perceived as an opportunity to gain commercial leverage rather than a practice in sound statistical validation.

In our view, there is a large gulf between the institutional cultures that support testing and validation of credit scores, and property-valuation models.

Among the least-appreciated aspects of property-valuation model testing and validation is the statistical concept of probability and expected outcome. This, in our estimation, is the issue that separates the process-intensive appraisal organizations from the risk-management culture that easily assimilated credit-risk scoring. The concept of probability and expected outcome is best explained in the context of testing.

A well-designed national test involves selection of a large sample of properties that is both reflective of the lending footprint as well as the product mix of the types of transactions that will, in the future, involve property-valuation models.

It is important to note that to be valid, a test must use transactions that are "out of sample"-meaning the test sample should not include transactions that have been used as part of model development. (A large national test conducted annually, for example, consistently selects its sample from public record data sources that are routinely used for model development. There is very little value in tests of this type.)

This is where a number of high-profile and influential tests break down. Often, they include publicly recorded sales transactions that have already influenced property-valuation models. Essentially, not conducting an out-of-sample test is akin to asking an 11-year-old child to correct his own math homework.

In a typical test, samples selected for testing involve properties that have either recent contract sale prices or appraised values-usually referred to as benchmark values. The sample is appended with property values generated by the models being tested. Model performance is determined in part by how well a given property-valuation model predicts the benchmark values of appraisal or sales prices.

Analysis of the results involves calculating measures of central tendency and dispersion, such as the variance, or difference, between estimated and benchmark values. Figure 1 provides an example of such an analysis based on a national test and a sample size involving more than 13,000 properties.

We can make general statements about model performance in terms of probabilities or expected outcomes. Most commercially available models estimate a property's market value within 10 percent of its benchmark value in roughly 75 percent of the cases. In other words, there is a 25 percent probability that the estimated model values will be off by more than 10 percent.

There are a number of reasons why a property-valuation model may produce "outliers"-estimates that vary widely from benchmark values. Usually, outliers are a consequence of incomplete or poor data content on the subject property, including changes in property condition over time. Naturally, users want to know when a model produces an outlier. In aggregate terms we can state that a model is likely to produce an outlier a predictable percentage of times for a given sample and geography. It is impossible, however, to absolutely state whether or not a particular calculation should be considered an outlier.

Many users of property-valuation models struggle with this concept of probability and expected outcome. In credit-score or prepayment-score applications, we make similar statements about the likelihood'of loan defaults or prepayments at different score calibrations.

To circumvent the issue of uncertainty about the valuation results for a specific property (and, perhaps, to soothe the anxieties of an uneasy marketplace), vendors have developed calculations that are generically referred to as "confidence scores." Note: The term "confidence score" is not to be confused with the statistical term "confidence interval" or "interval estimate," which are a range of values that is likely to contain the true value of the population parameter based on observations from a sample.

There is very little empirical evidence that these scores actually tell us anything conceptually or statistically significant. Yet they are widely used to accept or reject model results. In one recent test, a national lender informed us that any valuation with a confidence score of less than 50 on a scale of 1 to 100 would be not be accepted. When questioned about why the limit was 50, the response was because that is the halfway mark.

In our view, use of confidence scores in this manner is an ill-conceived attempt to outsmart the laws of probability theory. To quote Albert Einstein, "As far as the laws of mathematics refer to reality, they are not certain, and as far as they are certain, they do not refer to reality."

In a recent analysis of confidence scores comparing the accuracy of valuation model results for three national vendors, we found the statistical relationship between so-called confidence scores and model variance to be extremely weak. This means that, for every transaction where a customer accepted or rejected a value estimate based upon a confidence score threshold, the customer would have done nearly as well making his or her determination using a probability outcome, such as the flip of a coin.

A May 2005 statement by the Office of Comptroller of the Currency (OCC), Credit Risk Management Guidance for Home Equity Lending, Bulletin 220, aptly summarizes the confusion over the validity of confidence scoring of individual property valuations: "Many AVM vendors, when providing a value, will also provide a 'confidence score,' which usually relates to the accuracy of the value provided.. .. Institutions should also establish the confidence levels that are appropriate for the risk."

Has the OCC concluded in its own testing that "confidence scores" have a statistical relationship with model accuracy? Or that accepting a valuation that has a low "confidence score" actually increases loss probability and severity? Sweeping statements like these made in the absence of empirical evidence only add to confusion in the marketplace.

Redesigning the collateral-valuation workflow process

In the early 19905, property-valuation models were positioned as tools capable of replacing human appraisers for a significant percentage of mortgage applications. As a result, the industry has tended to evaluate these models in the context of replacing the appraiser. Today, we know that property-valuation models cannot totally substitute for manual appraisals, even in areas of the country where the depth and quality of data allow for reasonable and satisfactory results.

The single most important drawback of property-valuation models is their inability to gauge the property's condition or to take into consideration buyer-seller motivation. For these reasons, valuation models are not suitable as stand-alone collateral-assessment tools in transactions where property value or condition plays a large role in determining the total risk profile of the loan.

Enlarge 200% Enlarge 400%

Although some sophisticated users have combined property-valuation models with property inspections, the industry is still a long way from a more sophisticated approach where various elements of collateral and credit are operationally and technologically channeled into a single workflow process. Property-valuation models are most useful and effective when they are combined with credit and borrower characteristics on a single technology platform. This allows for a more efficient decision-making process on the type of collateral-valuation method required, as well as the ability to riskprice the loan.

Figure 2 illustrates the types of variables that might be used to produce a mortgage score. With the notable exception of the GSEs and a handful of national lenders, valuation models, by and large, are deployed as stand-alone tools outside of loan origination and underwriting systems.

From an operational and technology standpoint, collateral-valuation processes should be tied more closely with other aspects of mortgage risk management. This, to some extent, would appease investors, rating agencies and regulatory bodies. It would also bring about a more sophisticated and mature approach to the validation and implementation stages of property-valuation models. With the exception of the GSEs and a handful of national lenders, to date, the industry's approach to model validation and implementation has been disappointing.

Final thoughts

There is no question that property-valuation modeling has permanently changed collateral-assessment practices for a large segment of mortgage lending. However, in order to fully reap the benefits of statistically based valuations, industry thinking must mature on two levels.

First, the industry as a whole should begin to evaluate and analyze property-valuation models in the same fashion that credit scores were evaluated two decades ago. Proper statistical tools must be employed to analyze data and to establish credibility with the investors, rating agencies and regulators.

Second, the emphasis on innovation must shift from optimizing modeling methods to improving business practices employing property-valuation methodology. Property-valuation models are most promising when they are combined with credit and other borrower characteristics.