In July 2014, attorney Maura Grossman and professor Gordon Cormack introduced a new protocol for Technology Assisted Review that they showed could cut review time and costs substantially. Called Continuous Active Learning (“CAL”), this new approach differed from traditional TAR methods because it employed continuous learning throughout the review, rather than the one-time training used by most TAR technologies.
Their peer-reviewed research paper, “Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery,” also showed that using random documents was the least effective method for training a TAR system. Overall, they showed that CAL solved a number of real-world problems that had bedeviled review managers using TAR 1.0 protocols.
Not surprisingly, their research caused a stir. Some heralded its common-sense findings about continuous learning and the inefficiency of using random seeds for training. Others challenged the results, arguing that one-time training is good enough and that using random seeds eliminates bias. We were pleased that it confirmed our earlier research and legitimized our approach, which we call TAR 2.0.
Indeed, thanks to the Grossman/Cormack study, a new star was born.
How Does CAL Work?
CAL turns out to be much easier to understand and implement than the more complicated protocols associated with traditional TAR reviews.
TAR 1.0: One-Time Training
A Typical TAR 1.0 Review Process
A TAR 1.0 review is typically built around the following steps:
- A subject matter expert (SME), often a senior lawyer, reviews and tags a sample of randomly selected documents to use as a “control set” for training.
- The SME then begins a training process using Simple Passive Learning or Simple Active Learning. In either case, the SME reviews documents and tags them relevant or non-relevant.
- The TAR engine uses these judgments to build a classification/ranking algorithm that will find other relevant documents. It tests the algorithm against the control set to gauge its accuracy.
- Depending on the testing results, the SME may be asked to do more training to help improve the classification/ranking algorithm.
- This training and testing process continues until the classifier is “stable.” That means its search algorithm is no longer getting better at identifying relevant documents in the control set.
Even though training is iterative, the process is finite. Once the TAR engine has learned what it can about the control set, that’s it. You turn it loose to rank the larger document population (which can take hours to complete) and then divide the documents into categories to review or not. There is no opportunity to feed reviewer judgments back to the TAR engine to make it smarter.
TAR 2.0: Continuous Active Learning
In contrast, the CAL protocol merges training with review in a continuous process. Start by finding as many good documents as you can through keyword search, interviews, or any other means at your disposal. Then let your TAR 2.0 engine rank the documents and get the review team going.
Continuous Active Learning: A TAR 2.0 Process
As the review progresses, judgments from the review team are submitted back to the TAR 2.0 engine as seeds for further training. Each time reviewers ask for a new batch of documents, they are presented based on the latest ranking. To the extent the ranking has improved through the additional review judgments, reviewers receive better documents than they otherwise would have.
How Much Can I Save With CAL?
Grossman and Cormack compared CAL across eight different matters against two leading TAR 1.0 protocols (Simple Passive Learning (SPL) and Simple Active Learning (SAL). Without exception, they found that CAL allowed them to find relevant documents more quickly than either of the traditional approaches:
On average across all matters, CAL found 75% of the relevant documents in the collection after reviewing just 1% of the total documents. In contrast, a review team would have to look at over 30% of the collection with SPL (which relies on random sampling for TAR training). They would have to review about 15% of the collection with SAL (which lets the computer select some of the documents for training).
Using typical review rates and charges, CAL saved from $115,000 to as much as $257,000 over TAR 1.0 protocols. The time saved from having fewer documents to review is equally compelling, especially when facing tight discovery deadlines.
Solving Real-World Problems
Along with substantial cost and time savings, the CAL protocol solves several other real-world problems that have hindered the wider use of TAR.
1. Continuous Learning Yields Better Results
TAR 1.0 protocols are built on one-time training. Once that training is complete, the system has no way to get smarter about your documents. The team simply has to review all the documents ranked above the initial cutoff.
Grossman and Cormack show that CAL trumps one-time training. As the review progresses, the TAR 2.0 algorithm keeps learning about your documents. The team finds relevant documents more quickly, which saves substantially in time and review costs.
2. More Efficient than Random Training
TAR 1.0 required that you train the system using randomly selected documents. Proponents argued that if attorneys select training documents by other means (keyword searching, for example), their unwitting judgments could bias the system.
Grossman and Cormack showed that random is the least effective approach for TAR training. Where relevant documents are few and far between (often the case with review collections), the SME may have to click through thousands of documents before finding enough relevant ones to train the system. That results in slower and more costly review.
To address bias concerns, TAR 2.0 systems include random or specially selected documents (e.g. contextually diverse) in the review batches to ensure the review goes beyond the initial training documents. This way, the team spends most of its time reviewing highly relevant documents, while the system ensures they see enough other documents to make sure the review is complete.
3. Eliminates Need for Subject Matter Experts.
TAR 1.0 required that an SME train the system before review could begin. Often, a senior lawyer would spend hours reviewing thousands of documents before review could begin. Review managers chafed at waiting weeks for the SME to find time to review what were often irrelevant or marginal documents.
Grossman and Cormack showed that review-team training in a CAL protocol is far more effective. CAL systems base training on all the documents that have been reviewed, which means that natural variations by reviewers will not have a significant impact. As part of a quality control process, TAR 2.0 systems can present outliers to an SME for correction. Using reviewers to train the system makes the review cheaper (since experts bill at higher rates). It also lets review start right away, without waiting for the busy expert.
4. Seamlessly Handles Rolling Uploads
One impractical limitation of TAR 1.0 was the need to collect all documents before beginning training. Because early systems trained against a randomly selected control set, the set had to be selected from all referenced documents in order to be valid. If you later received additional documents, the control set was no longer valid. In most cases you had to start the training over again from scratch.
By contrast, CAL systems continuously rank all of the documents in your collection. When you add new documents, they simply join in the ranking process. Since training/review is continuous, new documents are quickly integrated along with the others in the ranking mix.
5. Excels at Low Richness Collections
TAR 1.0 systems choke with low-richness collections. One reason is the requirement that you train using randomly selected documents. When richness (the percentage of relevant documents in the collection) is low, it is hard to find relevant examples for training. Your SME may have to click through tens of thousands of documents. In some cases, the system won’t work at all.
CAL systems excel at low-richness collections, Grossman and Cormack concluded, often reducing the review population by 95% or more. This is because CAL starts by finding relevant documents through any means possible (keyword searches, witness interviews, analytics or otherwise). Those documents are fed to the TAR engine, which uses them to rank the document population. Reviewers get going right away reviewing the most-likely relevant documents first.
A Star is Born?
In the 1976 movie, an aging Kris Kristofferson helps newcomer Barbra Streisand break into show business. We quickly realize that her talent will soon eclipse that of the older star, whose career is in decline. Their love story is touching, but it doesn’t change the inevitable result. As the old star fades, the new one shines ever more brightly.
TAR is not show business, but there is a new star taking center stage. That star is Continuous Active Learning and it is pushing out the older TAR protocols. Grossman and Cormack have shown that it promises to save even more on review costs than its TAR 1.0 predecessors, while removing many of the limitations of those earlier protocols. A new TAR is born.