Removing Costs With Confidence

Over the past several months, record refinancing volumes—driven by the lowest mortgage rates in history—have led to an industrywide shortage of underwriters and overworked fulfillment teams, challenges that were exacerbated by the COVID-19 pandemic and the shift to remote work. With rates now rising, staff resources will need to be redistributed to address a growing share in a more labor-intensive purchase and servicing transfer market and cross-trained to enable greater business flexibility. Unfortunately, the hiring frenzy of 2020 may result in a resource surplus for 2021.

Now is the time to get serious about right-sizing high-value human skills and achieving scalability through automation, maximizing both time and money. Machine learning is a critical technology that is enabling both for the industry.

Measuring Confidence

What tasks can automation perform better than a human? Where is human oversight needed? How can the two work together to manage risk and optimize expense? In the mortgage industry, machine learning is successfully being used to achieve the right balance.

Some tasks that are always done the same way especially keystroke- and rules-driven—are ripe for automation. There is no grey area about whether the technology is performing the task correctly. Other tasks that are based on patterns and subject to changing variables are more challenging. This is the space where machine learning has been the breakthrough technology for automation.

What does machine learning have that other technologies don’t? Confidence. The degree to which the machine thinks it has “gotten it right” is known and quantifiable. That metric enables businesses to apply human oversight based on the importance and risk associated with the task. But before we get too far, let’s review some definitions.

First, it bears mentioning that confidence is not the same as accuracy. Within the context of loan automation technologies, accuracy refers to how often an automated process is correct in its outcomes. For example, if you run a pool of 100 documents through automated document recognition (ADR) technology, and ADR recognizes 89 documents correctly, that’s 89% accuracy.

Confidence, on the other hand, is the probability the result provided is in fact correct. In our example, the machine might have been 95% confident it got all 100 documents correctly classified. The 5% margin for error accounts for the misclassification of 11 documents.

If getting all 100 documents correctly classified is critical to a business, a human would need to do a quick review to verify each classification. On the other hand, if 95% confidence is good enough, a business can decide to “trust” the technology and eliminate human review altogether. The application of a confidence score becomes a useful guide for determining whether the results are likely to be right or wrong, and if that probability is acceptable for a given task.

Confidence scores are also helpful when determining if data being extracted from loan documents is correct. Here, algorithms can be applied to derive confidence scores of individual characters, and subsequently entire words or phrases. For example, if the data field is a street name, such as “Parker,” how confident is the machine that each character is what is reported, and how confident is the machine that the reported value of the entire street name is in fact “Parker”?

With this basic understanding of machine learning confidence scoring, let’s discuss in a bit more detail how lenders can apply it to their threshold settings to reflect their “comfort level” for data accuracy. The importance of each document and data element in a loan file varies across the loan lifecycle. The thresholds in new loan production help to ensure the manufacture of a quality loan. However, the same thresholds might not be applicable when selling the asset or servicing rights.

It would be expensive and time-consuming to review every document classification and data extraction again and again as a loan moves through production and due diligence reviews. Market participants rely on each other to implement the appropriate loan quality checks in each stage of the loan lifecycle, which in many instances is a regulatory requirement.

However, we all know mistakes do happen. This is where machine learning and confidence scoring can more efficiently enable the mortgage industry to manage loan quality holistically with less manual intervention.

While machine learning can automate all classification and data extraction at each stage, confidence scoring can enable each entity in the loan lifecycle to prioritize document and data accuracy through confidence scoring thresholds. The scores themselves can be used to help lenders control exceptions by dialing thresholds up or down, based on their specific requirements.

In particular, confidence scores can really help lenders gauge the quality of critical documents, such as loan applications or disclosures. By applying a higher confidence threshold to these documents, lenders can achieve greater safeguards for their accuracy.

The Value Behind Scores

By combining confidence scores with automated document classification and data extraction tools, lenders are better able to focus their resources on documents and data that demand greater scrutiny. This gets to the heart of value behind confidence scoring—it allows lenders to better control costs by minimizing the amount of human intervention.

For instance, a lender using ADR with confidence scoring may decide that all closing disclosure documents must have a score of 90% or higher in order to “pass,” and that those documents do not
need to be reviewed by a staff person. Basically, that’s money saved. Meanwhile, only the disclosures that fall below that threshold would trigger a review by a human processer or closer to confirm the document classification.

In the production space, confidence thresholds might be set high for key documents containing data used to underwrite the loan. Often these are first-generation documents, and the machine learning classification confidence is quite high, accurately classifying and extracting data with no human intervention.

In loan acquisition, whether acquiring closed loans or mortgage servicing rights (MSRs), loan file documents may be second or third generation in nature and much lower quality. This increases the likelihood that some critical documents evaluated in pre-purchase reviews may be classified but with low confidence. Setting a confidence threshold to “always fail” critical documents can ensure human review and greater transparency, which is important for due diligence.

Beware of Wild Claims

The proliferation of automated technology offerings in the mortgage industry has yielded a lot of unfounded claims by different providers. Many companies, for example, are fond of making blanket statements that their technologies deliver 95% (or whatever percent) accuracy. But what does this really mean? Does that percentage apply to all the different documents and document types that lenders encounter? Does it apply to all data fields, regardless of which documents they are found in? To what depth and to what breadth of mortgage life cycle use cases does this apply?

These grandiose claims over accuracy have proliferated for several years now. It’s easy to make such statements. But when you start peeling back the onion, you’ll find that most often accuracy only applies to a small set of common loan documents, not the vast library of documents in use today, and it becomes a very diminishing statement. As already mentioned, the “accuracy” of document processing technology is also highly dependent on the quality of the documents being processed—first, second, or third generation—which varies by channel.

Yet another consideration is what fail-safe measures—if any—have been put in place for handling exception scenarios. In other words, if a document cannot be classified, what method exists to review and update them? What is needed is some combination of exception management built into the technology that can complement human workflow to achieve the degree of accuracy required.

This is where the flexibility behind confidence scoring can truly make an impact. The ability to adjust confidence scoring thresholds for different document types and data points gives lenders the ability to reduce the “noise” created by different scenarios while maintaining high productivity and deploying staff resources in a more efficient manner.

The possibilities with confidence scoring are endless, but it’s important to remember nothing is 100% correct 100% of the time. Like any technology, confidence scoring is simply a tool that enables better decision-making. But when it comes to saving time and money, it’s a tool that can deliver high returns.