Catalyst Repository Systems - Powering Complex Legal Matters

E-Discovery Search Blog

Catalyst E-Discovery Search Blog RSS Follow Catalyst on Twitter Join Catalyst on Facebook Catalyst on LinkedIn Catalyst YouTube Channel
Follow Us:
Technology, Techniques and Best Practices

Court’s Suggestion to Use Predictive Coding Leads to Dispute over Cooperation

by Bob Ambrogi | June 6, 2013

Hat tip to K&L Gates Electronic Discovery Law blog for picking up on an interesting federal court opinion that, in the end, doesn’t actually decide anything substantive, but that is nonetheless notable for its illustration of some of the issues courts and counsel now face in the wake of last year’s groundbreaking opinion by U.S. Magistrate Judge Andrew J. Peck, Da Silva Moore v. Publicis Groupe, that was the first to give a judicial seal of approval to the use of predictive coding.

Perhaps most remarkable about this latest case, Gordon v. Kaleida Health, is that it was the judge, not the litigants, who suggested the use of predictive coding in the first place. Impatient with the parties’ year-long attempts to agree on how to achieve a cost-effective review of some 200,000-300,000 emails, U.S. Magistrate Judge Leslie G. Foschio of the Western District of New York pointed them to the Da Silva Moore ruling and suggested they try predictive coding. Here is how he explains it in his opinion:

At the last of a series of ESI discovery status conferences with the court, … the court expressed dissatisfaction with the parties’ lack of progress toward resolving issues related to completion of review and production of Defendants’ e-mails using the key-word search method, and pointed to the availability of predictive coding, a computer assisted ESI reviewing and production method directing the parties’ attention to the recent decision of Magistrate Judge Peck in Moore v. Publicis Groupe & MSL Group, 287 F.R.D. 182 (S.D.N.Y.2012), approving use of predictive coding in a case involving over 3 million e-mails.

It was a sensible suggestion on Judge Foschio’s part. Unfortunately, rather than bring the parties to agreement, the suggestion only gave them new grounds on which to disagree. Defendants took the judge up on his suggestion to use predictive coding, but the parties then quickly fell into disagreements over the extent to which plaintiffs would be involved in the predictive coding process. Plaintiffs wanted their ESI consultants to participate with defendants in establishing a protocol. They also wanted to meet with defendants to discuss various search issues that they believed were critical to the integrity of the process.

Plaintiffs Seek Agreement on Protocol

Defendants objected to having plaintiffs participate in establishing the protocol and to meeting to discuss search issues. Instead, defendants sent plaintiffs their protocol and said they would also send a list of their email custodians. When plaintiffs received defendants’ protocol, they objected to its use, asserting there were several technical issues that plaintiffs’ ESI consultants would be willing to discuss and help resolve. When defendants refused this request, plaintiffs filed a motion to compel defendants to meet and confer in order to establish an agreed-upon protocol for using predictive coding.

In support of their motion, plaintiffs contended that when a party intends to use predictive coding, it is necessary that the parties jointly negotiate a protocol to guide the process. In making this argument, they relied in part on the Da Silva Moore opinion, in which Judge Peck said, “Electronic discovery requires cooperation between opposing counsel and transparency in all aspects of preservation and production of ESI.” In keeping with this spirit of cooperation and transparency, they asserted, they should be privy to information regarding how defendants would select the set of seed documents that would be used to train the predictive coding engine.

In response to this motion, defendants told Judge Foschio they had never objected to meeting and conferring with plaintiffs regarding the predictive coding protocol. Rather, they had objected to meeting with a specific ESI consultant retained by plaintiffs, because the same consultant had previously provided services to the defendants in the same case.

Although now willing to meet with plaintiffs, defendants argued that the court should not require them to agree with plaintiffs on specific protocols, contending that the predictive coding process should be subject to the general rule that the method of ESI production is within the “sound discretion” of the producing party. Finally, defendants noted that the Da Silva Moore court never required the party using predictive coding to provide the seed set to its opponents; rather, the party volunteered to provide that data.

In the end, Judge Foschio seized on defendants’ representation that they were prepared to meet and confer with plaintiffs and plaintiffs’ ESI consultants, provided it was not the consultant who had worked for defendants. Given this representation, Judge Foschio said, there was no need to address other issues raised by plaintiffs’ motion. Thus, the judge dismissed the motion without prejudice.

The Bottom Line

Although the opinion decided no substantive issue, it is interesting for its discussion of the relative rights and obligations of litigants when one party chooses to use predictive coding. The opinion never decides the extent to which cooperation and transparency are required. But it outlines arguments we are certain to hear many times over again in future litigation.

As I said at the outset, the opinion is also interesting in that it was the judge who suggested predictive coding in the first place. Just a year ago, it was remarkable when a judge was willing to accept the litigants’ use of predictive coding. Now, we have a judge recommending its use of his own accord. That is a clear indication that, in terms of judicial acceptance of predictive coding, we’ve come a long way baby.

Filed Under: Predictive Ranking, Search Tagged With: , ,

Judge Tells Non-Party Google: Show Apple Your Search Terms

by Bob Ambrogi | May 29, 2013

E-discovery disputes typically arise between the parties to a lawsuit. But what happens when a non-party to the lawsuit is subpoenaed to produce electronically stored information? To what extent do its obligations mirror those that apply to the parties under the federal discovery rules?

Magistrate Judge Paul Grewal

That was the question in a recent case in which the non-party was none other than search giant Google. The question arose after another technology giant, Apple, as part of its ongoing patent litigation against Samsung, subpoenaed Google to produce certain documents.

After Google produced documents in response to the subpoena, Apple and Google held a meet-and-confer at which Apple raised concerns about the deficiency of Google’s production. To alleviate its concerns and enable it to evaluate the adequacy of the production, Apple requested that Google provide a list of the search terms and custodians Google used to find the documents.

Google refused to turn over the search terms and Apple filed a motion to compel. Earlier this month, U.S. Magistrate Judge Paul S. Grewal, sitting in the U.S. District Court in San Jose, Calif., issued his ruling on Apple’s motion.

Discovery Obligations of a Non-Party

As Judge Grewal explained, Google shifted in its arguments for why it should not turn over the search terms. Initially, at the meet-and-confer with Apple, Google maintained that its search terms and choice of custodians were privileged under the work-product immunity doctrine. It quickly abandoned this argument, the judge noted, “no doubt in part because case law suggests otherwise.”

Before the court, Google argued that producing the search terms and custodians would be unduly burdensome. The judge dismissed this in short order, explaining that Google provided no evidence to support this argument. Google also offered, in lieu of revealing its own search terms, to consider search terms and custodians suggested by Apple, but Apple declined.

After failing with these arguments, Google finally got Judge Grewal’s attention with the argument he described as “the heart of its opposition.” Google’s argument was that its status as a non-party to the lawsuit exempted it from the same sorts of obligations parties would bear to show the sufficiency of their production, at least until Apple first demonstrated that Google’s production was somehow deficient.

Google complains that “the impact of requiring non-parties to provide complete ‘transparency’ into their search methodology and custodians in responding to non-party subpoenas whenever unsubstantiated claims of production deficiencies are made would be extraordinary.” At the hearing, Google explained that providing custodians or search terms would open it to further burdensome discovery by Apple.

Judge Grewal, in his opinion, immediately rephrased Google’s argument in words that foreshadowed what he thought about it:

Google raises an important question: is it “extraordinary” to expect third parties to be transparent about their discovery methods? Underlying Google’s premise is that transparency in the discovery process is a burden or that the methods of discovery are somehow sacrosanct, and that revealing those methods opens the floodgates to more requests for discovery.

To resolve this question, Judge Grewal relied on a case that neither party cited in its arguments, DeGeer v. Gillis, 755 F. Supp. 2d 909 (N.D. Ill. 2010). It, too, involved a party’s request that a non-party provide search terms and custodians. And it involved a similar impasse, in which the non-party refused to turn over the terms and the party refused to suggest new ones. The DeGeer judge — now-retired U.S. Magistrate Judge Nan R. Nolan — ordered the non-party to produce the search terms and custodians in the hope it would facilitate meaningful discussions between the parties regarding any deficiencies in the production.

Although she ordered production, Judge Nolan was not happy with either side in the subpoena dispute. The non-party, by failing to disclose its search terms and custodians, “violated the principles of an open, transparent discovery process,” she wrote. At the same time, the party that sought the search terms had no excuse for its intransigence in failing to suggest search terms and custodians of its own, she added. Both sides should have cooperated in agreeing on search terms and custodians before the production ever took place, she said.

Admonishments for Both Sides

Judge Grewal found this reasoning applicable to the dispute between Google and Apple, concluding that each came up short in meeting its duty to collaborate with the other:

As the DeGeer court observed, transparency and collaboration is essential to meaningful, cost-effective discovery. Google’s attempt to stand outside of these tenets because of its third-party status is unpersuasive. Although it should not be required to “subsidize” litigation to which it is not a party, it confuses undue burden with its obligations, once subject to a subpoena, to participate in transparent and collaborative discovery. Third-party status does not confer a right to obfuscation or obstinacy.

Apple likewise failed to collaborate in its efforts to secure proper discovery from Google. It requested search terms and custodians only after it suspected that Google’s discovery was insufficient, and when Google offered to run additional terms on additional custodians, Apple made no effort to explore meaningful collaboration on obtaining the documents it believed were not produced.

Many years ago, when I was a young Catholic-school student, we would have called that brief lecture on cooperation a “slap on the knuckles.” But it remained for Judge Grewal to decide the question before him of whether to compel Google to produce its search terms and custodians. This he resolved in short order:

The court finds that production of Google’s search terms and custodians to Apple will aid in uncovering the sufficiency of Google’s production and serves greater purposes of transparency in discovery. Google shall produce the search terms and custodians no later than 48 hours from this order. Once those terms and custodians are provided, no later than 48 hours from the tender, the parties shall meet and confer in person to discuss the lists and to attempt to resolve any remaining disputes regarding Google’s production.

The ruling is barely seven pages but it is remarkably rich in the lessons it provides. For one, it makes clear that the e-discovery obligations of a non-party in response to a subpoena largely mirror those of a party under the federal rules. For another, its stands for the proposition that disclosure of search terms and custodians is consistent with the obligation that the e-discovery process be transparent and collaborative.

Last but not least, we cannot leave Judge Grewal’s opinion without noting the zinger of a footnote he dropped. As noted above, one of Google’s arguments was that producing its search terms and custodians would be unduly burdensome. After dispensing with this argument, Judge Grewal added the following in a footnote: “The court cannot help but note the irony that Google, a pioneer in searching the internet, is arguing that it would be unduly burdened by producing a list of how it searched its own files.”

Read Judge Grewal’s opinion here.

Filed Under: Case law, Search Tagged With: , ,

In Praise of Proportionality: Judge OKs Predictive Coding After Keyword Search

by Bob Ambrogi | April 29, 2013

Predictive coding purists might argue that the process is tainted if it is preceded by the use of keyword searching to reduce the document set. As a matter of fact, that was exactly what the plaintiffs argued in the multi-district litigation against Biomet over its M2a Magnum hip implant. But in a ruling last week, U.S. District Judge Robert L. Miller Jr. said that proportionality trumped purity, and that even if predictive coding might unearth additional relevant documents, the cost would far outweigh the likely benefits.

Emblème de la JusticeThe ruling came in a highly contentious matter in which Biomet has already produced 2.5 million documents out of a universe of 19.5 million. The plaintiffs say Biomet has not produced enough and that the production should total closer to 10 million documents.

Biomet began the process of producing documents in the summer of 2012, before the multiple lawsuits over its hip implant were centralized under the Judicial Panel on Multidistrict Litigation. It started by using keyword culling, reducing the number of documents from 19.5 million to 3.9 million, or 1.5 terabytes of data. Deduplication further reduced the number of documents to 2.5 million.

Biomet then employed predictive coding to identity relevant and privileged documents from among the 2.5 million that remained after keyword searching and deduplication. It also invited the plaintiffs to suggest additional search terms and offered to produce the rest of the non-privileged documents so that plaintiffs could verify that it was producing the relevant documents.

This process has cost Biomet $1.07 million so far and will end up costing between $2 million and $3.25 million, according to Judge Miller’s ruling.

The plaintiffs objected that Biomet’s initial use of keyword searching tainted the predictive coding process. They argued that keyword searching is less accurate than predictive coding, citing a recent article that said that keyword searches generate only a 20 percent responsive rate, compared to at least a 75 percent responsive rate for predictive coding. Because Biomet started with the less-accurate keyword searches, the entire process is flawed. The only remedy, they asserted, is for Biomet to go back to where it started and employ predictive coding against the original set of 19.5 million documents, with plaintiffs and defendants sitting together and jointly going through the process.

Balancing Cost against Benefits

Against this backdrop, Judge Miller framed the issue this way:

The issue before me today isn’t whether predictive coding is a better way of doing things than keyword searching prior to predictive coding. I must decide whether Biomet’s procedure satisfies its discovery obligations and, if so, whether it must also do what the Steering Committee seeks.

Judge Miller easily decides that Biomet’s process complied with the Federal Rules of Civil Procedure and with the Seventh Circuit Principles Relating to the Discovery of Electronically Stored Information, as well as with principles set forth by The Sedona Conference.

By contrast, he says that plaintiffs’ request that Biomet go back to Square One “sits uneasily with the proportionality standard in Rule 26(b)(2)(C).” The cost of this, he notes, would run in the “low seven figures,” but Biomet’s testing suggests it would find only a “comparatively modest number of documents.”

It might well be that predictive coding, instead of a keyword search, at Stage Two of the process would unearth additional relevant documents. But it would cost Biomet a million, or millions, of dollars to test the Steering Committee’s theory that predictive coding would produce a significantly greater number of relevant documents. Even in light of the needs of the hundreds of plaintiffs in this case, the very large amount in controversy, the parties’ resources, the importance of the issues at stake, and the importance of this discovery in resolving the issues,I can’t find that the likely benefits of the discovery proposed by the Steering Committee equals or outweighs its additional burden on, and additional expense to, Biomet.

Judge Miller concludes by stating that he assumes Biomet will remain open to meeting and conferring on additional, reasonable search terms and to producing the non-privileged documents included in the statistical sample. “Beyond that,” he writes, “if the Steering Committee wishes production of documents that can be identified only through re-commenced processing, predictive coding, review, and production, the Steering Committee will have to bear the expense.”

The full text of Judge Miller’s order is below. (If you are not able to see the embedded document below, click here to see it.)


Filed Under: Case law, Predictive Ranking, Search Tagged With: , , , ,

My Key Word Searches are Better than Your Predictive Ranking Technology!

by John Tredennick | January 23, 2013

I recently got a distress call from an e-discovery partner of ours with an unhappy client. “It seems like there is something wrong with your predictive ranking technology,” our partner said on the Google Hangout. “It’s proposing that the client team review too many documents–more than we got with key word searching. Our client is upset. We need to do something to explain this fast.”

In this case the client team had not used technology assisted review (TAR) before; this was their first try at the process. They wanted proof that it was worth the extra cost for the technology. Specifically, they wanted to see whether it actually cut down on review costs, like everyone claimed.

The problem was that the system didn’t seem to work–at least in their eyes. They had started the review process by running a series of key word searches, which was their normal practice. The searches hit on a total of about 11,000 documents, out of a test set of 50,000 documents. This suggested they had only 11,000 documents to review for the production, about 20% of the total collected.

Our partner had recommended that the client try our Predictive Ranking technology as a better means to find responsive documents. Everyone’s initial expectation was that doing so would reduce the review population even further than the 11,000 documents that the key word searches hit. Somehow they got the impression that the review population might go down to more like 7,000. That would certainly justify the extra expense of this new technology.

Unfortunately, the opposite turned out to be the case. Instead of recommending that the team review 7,000 documents, or even 11,000 documents, our system suggested reviewing more than 18,500 documents.

You can imagine the client’s consternation. “You want us to pay you for these results? You just increased rather than reduced my review costs. I like it better the old way.”

It was time to go to work to figure out what happened. Fortunately, our team had analyzed the key word search results and compared them to the documents identified through Predictive Ranking. My job was to explain the difference between the two approaches. My hope was to show that the system worked well and provided a better outcome than key word search¾at least if the goal was to identify and review potentially relevant documents.

To be sure, our Predictive Ranking system came up with more documents to review than did the key word searches. However, we quickly concluded that the key word searches–while finding many potentially responsive documents–missed a lot of others that should be considered as well. Here is how we reached that conclusion.

The Numbers

Let me start by giving you some basic information about the two processes. From there it becomes a bit easier to explain the difference in the results. You can then see how we got to our ultimate conclusion.

The client collected just over 51,000 documents for this production. As a first step, counsel created a set of key word searches and asked our partner to run them using our PowerSearch utility. As I mentioned earlier, the searches hit on about 11,000 documents.

We then started the Predictive Ranking process, which is the name we started using years ago for our TAR methods. The first step was to take an initial random sample for reference purposes and to estimate the overall richness of the population. We came up with an estimate of 22%, which would suggest that there were about 11,000 relevant documents in the collection.

Hmmm. That was pretty close to the number of documents found through the key word searches. Did they nail it this time?

We next worked with the client’s legal experts to review and tag seed documents and then to undertake several rounds of system training. At the end of the process, the system suggested a review cutoff below 36%. That meant that the review team would look at the top 36% of the documents and ignore (after a confirmatory sample) the remaining 64%.

The resulting numbers looked like this:

  • Likely responsive and need review: 18,552.
  • Likely non-responsive and don’t need review: 32,982

Our sampling also suggested that the top documents above the cutoff had a richness of about 50% (which meant half of these were likely responsive). The documents below the cutoff had a richness of about 7% (which meant that only 7 out of 100 were likely responsive). Seven percent seemed like a good number for the discard pile, one that most courts would accept.

The Question

As mentioned earlier, our results caused heartburn for our partner and its client. The key word search approach seemed to require that the team review only 11,318 documents. Why pay for Predictive Ranking if it requires that the team review 7,000 additional documents? That’s about 60% higher than the key word results.

Understanding the Numbers

The answer requires that we better understand what all these figures mean. Unless we can compare apples to apples, we have no way to judge the efficacy of the two approaches. Fortunately we had an easy way to do just that.

We compared the document IDs for the files returned from the key word searches with the files returned from our Predictive Ranking process. What we found was pretty interesting. Let me show you with this simple diagram:

I created this diagram to map the comparative results of our Predictive Ranking and the key word searches. The circle represents the total documents in the population. If you add up the numbers in the four quadrants, it comes up to the 51,534 files at issue

The four quadrants represent the different states of the document population.

  • The top left quadrant represents documents that our Predictive Ranking system found likely responsive but did not return from the key word searches. There were 11,285 documents in this category.
  • The top right quadrant represents documents that hit under both approaches. These documents were returned by the key word searches and were also designated by our Predictive Ranking system as potentially responsive. There were 7,267 documents in this category.
  • The bottom left quadrant represents documents that did not hit under either approach. Neither our Predictive Ranking system nor the key word searches deemed them likely responsive. There were 28,931 documents in this category.
  • The bottom right quadrant represents documents that were returned from the key word searches but were not deemed likely responsive by our Predictive Ranking system. There were 4,051 documents in this category.

So, what can we say about all this? First, we can say that the team should probably review the 7,267 documents found in the top right quadrant. Both approaches tagged them as likely responsive. That does not mean that they will all be responsive but it is a good bet that a lot of them are.

Second, we can suggest that the 28,931 documents in the bottom left quadrant  include few responsive documents. Neither the key word searches nor our Predictive Ranking system hit on these documents. There is still a need for confirmatory sampling but we can be pretty sure that there are not a lot of responsive documents hiding in this quadrant.

The Two Key Quadrants

That leaves us with two quadrants to consider and this is where we find the answer to our puzzle. Together, these two quadrants represent about 15,000 documents. Here is what we can say about each:

  • The top left quadrant represents 11,285 documents that our Predictive Ranking system found as likely responsive. The keyword searches provide no information about these documents other than to say that they did not return from the searches.
  • The bottom right quadrant represents 4,051 documents that hit on counsel’s keyword searches but our Predictive Ranking system found to be likely non-responsive.

If counsel only reviewed documents that returned from the key word searches, they would be ignoring the 11,285 documents identified in the top left quadrant. Many of them had already been tagged during the Predictive Ranking training session and thus we knew that there were responsive documents in this quadrant. Our richness estimate went so far as to suggest that 50% of them were likely responsive, which meant that counsel might be missing 5,000-6,000 responsive documents using their key word approach. It quickly became evident that counsel would have to at least test additional documents in this quadrant before dismissing them as not responsive.

Conversely, our Predictive Ranking system led us to question how many of the 4,051 documents in the lower right quadrant were responsive. In fact, we knew from training that many of the documents in that quadrant were not responsive. At the least, that is what the reviewers concluded when they addressed them during the sampling.

We suggested that the client test the documents in this quadrant before engaging in review. Our suspicion was that these were false hits from the key word searches and not likely of interest. Our estimate was that they would find a richness of about 7%.

Answering the Question

By now you have already figured out how to respond to the client’s concerns. Simply put, the key word searches–while effective at finding some of the potentially responsive documents–missed a lot of others that should be reviewed. The Predictive Ranking system found many of the documents returned from the key word searches but it also found a lot of other potentially responsive documents. The total numbers were higher but there was good reason for that outcome. There were more documents that needed to be reviewed.

Put another way, search has two qualitative measures: precision and recall. Precision is a measure of the number of true hits (actually responsive documents) returned from your search compared to the total number returned. Recall is a measure of the total true hits returned from your search against the actual number of true hits in the population.

In our case, the key word searches may have been good on precision (assuming that the documents in the top right quadrant were, in fact, responsive). However, they seemed to miss the boat on recall. The searches missed a lot of the other responsive documents. That is not a good thing if your opponent chooses to challenge your production in court.

It turned out that my explanation proved helpful to our partner and the client team. They moved forward with their review using the documents ranked by our Predictive Ranking system. It turned out that there were a lot of responsive documents missed by the key word searches and many of the documents returned by the searches in the lower right quadrant were false hits. The explanation and diagram helped to clear up the mystery. I thought it might be helpful to others as well as they grapple with the mysteries of technology assisted review.

There may also be a moral to this story, so to speak. Discussion of technology assisted review often focuses on its ability to reduce document populations. But review is not just a numbers game—it’s also about getting it right. It does neither lawyers nor their clients any good to cut document populations if they are cutting a large number of potentially responsive documents in the process. As my story above illustrates, fewer is not always better. Here, Predictive Ranking proved itself superior to key word searching at getting it right. That may have saved counsel some grief further down the road.

Filed Under: Predictive Ranking, Review, Search Tagged With: , ,

Treading Past Angels: Finding the Right Search Expert for Your Case

by Ron Tienzo | January 15, 2013

Last February, Assured Guaranty Municipal Corp. sued UBS Real Estate Securities Inc. for breach of contract, accusing the company of failing to meet obligations related to the pooling of residential mortgage-backed securities. As the case moved along, disputes arose over discovery and both sides filed motions to compel.

One of the discovery issues in dispute was the adequacy of the search terms that Assured proposed to apply to electronic documents. Ruling on this issue in a Nov. 21, 2012, memorandum, U.S. Magistrate Judge James C. Francis IV began by quoting the oft-cited words of another U.S. magistrate judge, John M. Facciola, in U.S. v. O’Keefe:

Whether search terms or ‘keywords’ will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. … Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.

Because neither party provided the expert affidavits that would have enabled him to decide on the most efficient search protocol, Judge Francis laid out three options for counsel:

  1. They can cooperate (along with their technical consultants) and attempt to agree on an appropriate set of search criteria.
  2. They can refile a motion to compel, supported by expert testimony.
  3. Or, they can request the appointment of a neutral consultant who will design a search strategy.

Note that in each of the three options, Judge Francis requires that an expert be involved in any determination, even when both counsel come to an agreement out of court.

So, if counsel are required to consult an expert and if search creation and testing is truly an area where “angels fear to tread,” how do you make sure to select the right search expert for your case? Below are some tips you may find helpful in finding the right consultant.

E-Discovery Expert  ≠ Search Expert

Electronic discovery is an extremely broad field of knowledge covering the entire gambit of the EDRM [link: http://www.edrm.net/resources/edrm-stages-explained]. While there may be a few e-discovery experts out there qualified to serve as search experts, Judge Franklin makes clear that the expert should have a solid understanding of the interplay between “the sciences of computer technology, statistics and linguistics.”  In the world of academia, this field of study is referred to as “Information Retrieval” (IR). [link: http://en.wikipedia.org/wiki/Information_retrieval] Those specializing in IR seek to use searches to identify the relevant information from a larger sea of data. When selecting your expert, be sure she has expert knowledge in at least one of the areas of expertise listed by Franklin and foundational knowledge of the others. She should be able to understand how each science plays into the ultimate goal of creating defensible searches.

One Size Does Not Fit All

Like electronic discovery, IR is an extremely broad area of study. In 1992, the Text Retrieval Conference (TREC) [link: http://trec.nist.gov/ ] was created to support the IR community by providing the infrastructure to test and evaluate different text retrieval methodologies.  Since its inception, there have been 21 different research areas, or tracks. These tracks range from web search to spam filtering to gene sequencing. While the Legal Track has been a mainstay for TREC since 2006, it is far from the only IR area of study.

Much of IR focuses on returning the best results from an extremely large dataset. A good example of this would be web search. In web search, the user enters a query and the IR system compiles a list of matching results in a ranked order by likelihood or responsiveness. Users typically find what they are looking for within the first few pages of results and have no need to proceed further.

In contrast, an ideal e-discovery search methodology would need to make an accurate relevance assessment on each and every document in the collection. Instead of finding just the best documents, we need to identify those with even the slightest traces of relevant content. This task poses a unique and what I believe to be a significantly harder task than the majority of IR research.

When selecting a search expert, be sure to find one who is familiar with the intricacies of legal search. While the ideal candidate does not need to have a JD, the expert should be familiar with what is required under the Federal Rules of Evidence and be able to conform her processes to that standard.

Those Who Can, Do; Those Who Can’t, Teach; Those Wanting to Be My Expert, Do and Teach!

It’s obvious that you want to find an expert who has done similar work before and has the references to confirm it. Supporting documentation can be in the form of scholarly articles, prepared client reports, and documents submitted to the court. You want to read through these documents to make sure the information is presented clearly, that a structured workflow was followed, and the results are supported by iterative testing. You also want to make sure that throughout the expert’s career, she has been consistent in her work. Prior publications represent a minefield for prior inconsistent opinions and will likely be used by opposing counsel to discredit your expert

You also want to find an expert who can clearly explain the craft. Most experts are used to operating in a highly specialized academic community with shared foundational knowledge and common vocabulary. A good expert needs to be a teacher at heart. She should enjoy educating others by explaining complex principles in simplified terms.

Have the expert walk you through one of her reports. Make sure she can detail in basic vocabulary each step that was taken, why choices were made, and how the outcome was determined. For any part of the process that requires foundational knowledge, such as statistical sampling, she should be able to break down the process enough that the user can follow along and then return to the bigger picture. Be sure to ask questions, keeping a close eye to make sure she does not appear frustrated or inconvenienced.

Our Catalyst Consulting team of linguists, statisticians, attorneys, and technical experts has been helping clients construct repeatable and defensible search methodologies since 2008. For assistance with your case, or to find out more on selecting the perfect expert, be sure to contact them at consulting@catalystsecure.com.

Filed Under: Case law, Search, Uncategorized Tagged With: ,

In Search, Evaluation Drives Innovation; Or, What You Cannot Measure You Cannot Improve

by Jeremy Pickens | October 19, 2012

Information retrieval researchers at Shonan last week. That's me in the center, wearing the yellow T-shirt.

Last week, I was honored to join a small group of information-retrieval researchers from around the world, from both industry and academia, who gathered at the Shonan Village Center in Kanagawa, Japan, to discuss issues surrounding the evaluation of whole session, interactive information retrieval. In this post, I introduce the purpose of this meeting. In later posts, I hope to further review the discussions that took place at Shonan and my own impressions.

Traditionally, information retrieval (a.k.a. search) has been viewed as a stateless, non-interactive process. The user issues a(n ad hoc) query to a search engine and the engine responds with its best attempt at answering that query, with results ranked by their likelihood of satisfying a user’s information need.

Interactive information retrieval, on the other hand, presumes multiple rounds of user-system exchange. The interactions during this exchange are presumed to be non-independent. Each query has some sort of relationship to previous queries, if only because the overall series is in support of the same user task or goal.

Examples of scenarios in which interactive information retrieval is necessary include travel or event planning, education and learning, seeking entertainment, and (of course) e-discovery. When queries are independent, the best the system can do is answer each query as if it were the last that the user will ask. However, when queries are non-independent, both the user and the system have the chance to engage in deeper and wider patterns of exploration.

Evaluating Interactive Information Retrieval

Evaluation of one-shot queries has a long and rich history. Concepts such as “binary relevance” and “precision and recall,” combined with batch mode evaluation, have led to countless advances in the state of the art. These advances, from the 1960s to the 1990s, allowed search engines, especially in a web context, to improve to the point at which they now bring huge benefits to society. Evaluation of interactive information retrieval tasks, on the other hand, does not have as-yet universally accepted metrics. The very nature of the interactivity (non-independence of a sequence of user actions and system responses) both gives the scenario its power and makes it difficult to evaluate.

The power, again, comes from the breadth and depth of what is made possible; the evaluation difficulty by this very same interdependence. When a single query is performed, it can be generally expected that the user traverses the results list in linear order, from estimated best to estimated worst result. And, with some probability, the user abandons the list traversal. These (generally realistic) assumptions allow the ad hoc, one-shot query to be evaluated in terms of the position of relevant documents within the list.

However, when multiple queries are performed, an element of non-determinism enters into the picture. A user typically does not examine all results in the list from the first query, then all results in the list from the second query, and so on. Instead, one user might only examine 57 results from the first query, 9 results from the second query, and then 82 results from the third query. Another user might examine 3 results from the first query, 18 results from the second query, and 17 results from the third query.

Furthermore, the order in which the results are seen by the user affects the next round of interactivity. That is, the second and third queries that are issued by an information seeker are influenced by which documents were seen during the first round of interaction. Even if two users started with the same first query, the user who looked at 57 results might have a very different notion of how to formulate the next query than the user who looked at only 3 results.

How, then, should these two users’ experiences with the interactive search engine be evaluated? Should it be the product or sum of the quality of the individual ranked lists for each query? That ignores the depth to which the user actually traveled in each list over the course of the session. Should evaluation instead be a function of the sequence of documents that the user actually saw during the course of the session, no matter which individual results list a document came from? That is better, but it still ignores the effects of document examination order on the queries that were issued — and more importantly on the queries that could have been issued, had the user traversed to either a shallower or deeper position within a particular list. The non-deterministic range of possibilities poses a severe challenge to the evaluation of interactive information retrieval.

Another issue related to whole-session evaluation in interactive information seeking has to do with progress during versus upon completion of an entire session. Should the primary focus of evaluation be to estimate the quality of a session only at the end of the user’s sequence of interactions? Or is it more important to have a metric which measures, i.e. expects, progress throughout a session? Inherent in the answer to this question is whether one expects interactive information retrieval progress to be linear. Is it? Should it be? The answer is an open question, one which we discussed at the Shonan Meeting.

Evaluation drives innovation. If you cannot measure something, you cannot improve it. The first step to improving interactive information retrieval systems is knowing what to measure and how to measure it. Only then will consistent improvements be possible.

Filed Under: Search Tagged With: ,

What Google Can Teach You about Effective E-Discovery Search

by Bob Ambrogi | September 14, 2012

In e-discovery, it all comes down to search. All the time spent collecting and reviewing. All the whiz-bang platforms from an array of vendors. All the newfangled technologies such as predictive coding and computer-assisted review. They all have one predominant purpose: To search for nuggets within mountains of data.

As Catalyst’s CEO John Tredennick put it so well in a post here last year, “Without search, we would be in a world of hurt, at least for e-discovery.”

Given this, it is surprising how little so many of us understand about even the most rudimentary principles of effective search. Google has spoiled us. We type some words in the query field and expect results. That may work for finding a recipe or a restaurant, but it doesn’t stand up to standards of defensibility in e-discovery.

What many e-discovery professionals desperately need is a course in the fundamentals of basic and advanced search techniques. It should be a course that would cover the key concepts, that would explain how to use operators, that would show how to filter results, and that would demonstrate how to tie these together to create complex searches.

Ideally, because we’re all busy, the course could be taken from our desktops, at our convenience. And, because we’re all tightwads at heart, the course should be free of cost.

As it turns out, there is such a course, and it comes courtesy of the very search powerhouse that I’ve already blamed for spoiling us, Google.

Power Searching with Google

In July, Daniel Russell, a senior research scientist at Google, presented a course, Power Searching with Google. The course consisted of a series of videos of Russell demonstrating search techniques, along with activities and assessments you could perform to test your skills. Those who completed it were awarded a “Power Searching with Google” certificate. Until just a few days ago, the entire course remained online, available for anyone to view. The July course is now closed, but the good news is that Google is about to begin a new session. It starts Sept. 24 and registration remains free of charge.

The course is composed of six “classes,” each consisting of multiple lessons. From start to finish, the classes take you from elementary to advanced, beginning with a basic discussion of how search works and ending with demonstrations of how to combine advanced techniques to find pinpoint results.

Let me be clear: This is not a class in e-discovery search. It is about Google search. However, while the search syntax the lessons use is specific to Google, the search concepts they teach are universal.

In fact, it was one of the search consulting experts at Catalyst who told me about the course. “Every law student and every practicing attorney embarking on his or her first intensive e-discovery case should take this course,” he urged.

Here is some of what the course teaches:

  • Class 1 is an introduction to “being a great internet searcher.” It covers how search works, the art of keyword choices, and why word order matters, among other topics.
  • Class 2 is a series of lessons on how to interpret and build on search results.
  • Class 3 begins to explore advanced search techniques, demonstrating ways both to filter and expand a search and to remove invasive or irrelevant results.
  • Class 4 is titled “Find facts faster” and it focuses on features built in to Google that help you narrow and refine your searches and quickly jump to specific types of results.
  • Class 5 covers fact checking and includes tips worth noting in e-discovery, such as how to search for variations of a concept and how to avoid biasing your searches through your queries.
  • The final class consists of three lessons focused on “putting it all together.” It shows you how to combine search techniques into complex queries and how to “think outside the box” in constructing searches.

The lessons are presented through a series of short videos that you can watch in bite-sized chunks. Text versions of each lesson are also available. I recommend both: Watch the videos then save the documents for later reference.

If the course whets your appetite for search information, you may also want to follow Russell’s blog, SearchReSearch, where he writes about search and search skills and offers regular search challenges where you can test your own search skills. Also, Google provides a range of training videos on everything from Google Scholar to Google Maps through its Google Search Education site.

As that Catalyst search expert said to me in recommending this course, “Not only will you be a better searcher, you may just impress your kids next time they ask you for help on their science project.” That alone is worth its weight in nuggets.

Filed Under: Search, Seminars, Tips and Training Tagged With: ,

The Next Big Predictive Coding Case that Wasn’t

by Bob Ambrogi | August 30, 2012

The case that many believed might be the next big bang in predictive coding jurisprudence instead has ended with barely a whimper.

As I noted here last month, in the wake of Magistrate Judge Andrew J. Peck’s ruling in Da Silva Moore v. Publicis Groupe affirming the use of predictive coding, many in the e-discovery field turned their attention to Kleen Products LLC v. Packaging Corporation of America, believing that it might be the Next Big Case on predictive coding.

The plaintiffs in Kleen Products had asked U.S. Magistrate Judge Nan Nolan to require the defendants to use predictive coding and Judge Nolan had conducted two days of evidentiary hearings on the request as well as several status conferences.

Although the case continues on, predictive coding is off the table, at least for the time being. Last week, Judge Nolan approved a stipulation submitted by the parties in which plaintiffs withdrew their demand to apply predictive coding to any documents relating to any request for production filed prior to Oct. 1, 2013.

As to any requests for production filed after that date, the parties stipulated that they will meet and confer regarding the appropriate search methodology. “If the parties fail to agree on a search methodology,” the stipulation says, “either party may file a motion with the Court seeking resolution.”

That suggests that we may not have heard the last of Kleen Products in the context of computer-assisted search. But with any further possible rulings on the issue well over a year away, we can safely write it off as the next big case.

Filed Under: Case law, Predictive Ranking, Search, Uncategorized Tagged With: , ,

Court Orders Counsel to Disclose E-Discovery Search Strategy

by Bob Ambrogi | August 28, 2012

Out of concern that counsel may not have sufficiently supervised their client’s production of electronic documents, a federal judge in New Mexico has ordered the attorneys to disclose the search strategy their client used to identify responsive documents. In so ruling, the judge relied on the federal rule that requires attorneys to sign discovery responses and certify that they are “complete and correct.”

Addressing motions to compel discovery in the case of S2 Automation LLC v. Micron Technology, U.S. District Judge James O. Browning ruled that S2 Automation would have to provide to Micron “its search strategy for identifying pertinent documents, including the procedures it used and how it interacted with its counsel to facilitate the production process.”

The judge based his ruling on Federal Rule of Civil Procedure 26(g), which requires that discovery responses must be signed by attorneys who must certify that the response “is complete and correct as of the time it is made.” Judge Browning concluded that this certification obligation is analogous to the certification required under Fed.R.Civ.P. 11 and that case law interpreting Rule 11 applies to Rule 26(g).

Applying that case law to the discovery context, Judge Browning said that Rule 26(g) imposes an obligation on the attorney who signs the discovery response to conduct “a reasonable inquiry into the facts and law supporting the pleading.” He went on to explain:

Accordingly, it can become necessary to evaluate whether an attorney complied with his rule 26(g) obligations and to evaluate the strategy an attorney used to provide responsive discovery, with relevant circumstances including: (i) “[t]he number and complexity of the issues”; (ii) “[t]he location, nature, number and availability of potentially relevant witnesses or documents”; (iii) “[t]he extent of past working relationships between the attorney and the client, particularly in related or similar litigation”; and (iv) “[t]he time available to conduct an investigation.” 6 J. Moore, Moore’s Federal Practice, § 26.154[2][a], at 26-615 (3d ed. 2012). Consequently, the analysis in which courts must engage to evaluate whether a party’s discovery responses were adequate is often a fact-intensive inquiry that requires evaluation of the procedures the producing party adopted during discovery.

‘Not Proper for Counsel to Sit Back’

The issue came before the judge after Micron’s attorneys became aware that S2′s counsel may not have worked with their client sufficiently during the discovery process and may have failed to provide a number of responsive documents. They became aware of that during a discovery conference between the parties, which followed a deficiency letter Micron sent S2 detailing shortcomings in S2′s production. A sworn declaration from one of Micron’s attorneys detailed what allegedly happened during that conference:

During that call, we discussed the April 25 deficiency letter and Micron’s request that S2 supplement its production. Counsel for S2 stated that he had not yet reviewed the letter in detail. We then discussed the format for production of S2′s documents. Counsel stated that he was not aware that S2 had separated attachments from e-mails, that he had delegated the process of gathering documents to S2, and that he was generally unaware of the manner in which S2 had provided the documents. Counsel also stated that he was unsure what protocol S2 followed to locate responsive documents.

In its motion to compel, Micron argued that “it is not proper for counsel to sit back and allow the client to search for documents without active direction and participation by counsel; to the contrary, counsel must be actively involved in the search to ensure that all responsive documents have been located, preserved, and produced.” The approach taken by S2′s counsel violated their obligations under Rule 26(g), Micron asserted.

S2′s attorneys denied that they had failed to supervise the discovery process, asserting that “nothing could be further from the truth” and that they had “met with the client on multiple occasions during the discovery process in order to organize and respond to discovery.”

Despite S2′s protestations, Judge Browning wrote that Micron’s sworn declaration caused him to be concerned about the adequacy of S2′s strategy for responding to discovery requests. “The … Declaration suggests that S2 Automation’s counsel were not working closely with their client during the document-production process,” he said. “Without some information about the search strategy S2 Automation used to provide responsive documents to requests for production, neither the Court nor Micron Technology can have a full understanding of the adequacy of S2 Automation’s search strategy.”

For this reason, Judge Browning concluded that it was appropriate for him to order S2 to provide to Micron its search strategy for identifying pertinent documents. His order directed S2 to include explanations of the procedures it used and how it interacted with its counsel to facilitate the production process.

What Can We Learn from this Case?

This case serves as a reminder that the lawyer who signs a discovery response has a direct obligation under Rule 26(g) to ensure that the response is complete and correct. As Judge Browning indicated, the key inquiry is whether the lawyer who signed the response “conducted a reasonable inquiry into the facts and law supporting the pleading.”

In this case, the judge’s uncertainty about whether the lawyers lived up to that obligation resulted in his order that their client explain its search strategy. In appropriate cases, however, Rule 26(g) also authorizes courts to impose sanctions on the lawyer and the client. “The sanction may include an order to pay the reasonable expenses, including attorney’s fees, caused by the violation,” the rule says.

Filed Under: Case law, Sanctions, Search Tagged With: , , ,

Courts Should Consider Search Technology, Say New Penn. E-Discovery Rules

by Bob Ambrogi | June 18, 2012

The Supreme Court of Pennsylvania

The Supreme Court of Pennsylvania has adopted new e-discovery rules that expressly distance federal e-discovery jurisprudence and instead emphasize “traditional principles of proportionality under Pennsylvania law.” Notably, the new rules provide that, when weighing proportionality, parties and courts should consider electronic search and sampling technology, among other factors.

The court promulgated the new e-discovery rules June 6 as amendments to the Pennsylvania Rules of Civil Procedure. They take effect Aug. 1, 2012.

The most significant change is to Rule 4009.1, governing requests for the production of documents and things. The current rule defines “documents” as including:

electronically created data, and other compilations of data from which information can be obtained, translated, if necessary, by the respondent party or person upon whom the request or subpoena is served through detection or recovery devices into reasonably usable form.

The amendment deletes this entire phrase and replaces it with the simpler phrase, “electronically stored information.” The amended rule will now read:

Any party may serve a request upon a party pursuant to Rules 4009.11 and 4009.12 or a subpoena upon a person not a party pursuant to Rules 4009.21 through 4009.27 to produce and permit the requesting party, or someone acting on the party’s behalf, to inspect and copy any designated documents (including writings, drawings, graphs, charts, photographs, and electronically stored information), or to inspect, copy, test or sample any tangible things or electronically stored information, which constitute or contain matters within the scope of Rules 4003.1 through 4003.6 inclusive and which are in the possession, custody or control of the party or person upon whom the request or subpoena is served, and may do so one or more times.

But while the rule adopts the phrase used in the federal rules, the official comment makes clear that the court’s intent is not to adopt federal e-discovery law:

Though the term “electronically stored information” is used in these rules, there is no intent to incorporate the federal jurisprudence surrounding the discovery of electronically stored information. The treatment of such issues is to be determined by traditional principles of proportionality under Pennsylvania law as discussed in further detail below.

One other significant change to the rule is addition of a new subparagraph (b) to Rule 4009.1 which addresses the form of production. The new rule says that the party requesting ESI may specify the format in which it is to be produced, to which the responding party may object. If the requesting party does not specify a format, then the ESI may be produced “in the form in which it is ordinarily maintained or in a reasonably usable form.”

Proportionality Should Prevail

The official comment to the amended rules emphasizes the importance of proportionality in determining the scope of discovery obligations. The overarching goal of the rules, the comment says, is to ensure that discovery is conducted in a manner that is “consistent with the just, speedy and inexpensive determination and resolution of litigation disputes.” To that end, the comment continues, courts faced with discovery disputes should consider five factors:

  1. The nature and scope of the litigation, including the importance and complexity of the issues and the amounts at stake.
  2. The relevance of ESI and its importance to the court’s adjudication in the given case.
  3. The cost, burden, and delay that may be imposed on the parties to deal with ESI.
  4. The ease of producing ESI and whether substantially similar information is available with less burden.
  5. Any other factors relevant under the circumstances.

The comment goes on to identify what it describes as “tools for addressing” ESI. It says:

Parties and courts may consider tools such as electronic searching, sampling, cost sharing, and non-waiver agreements to fairly allocate discovery burdens and costs. When utilizing non-waiver agreements, parties may wish to incorporate those agreements into court orders to maximize protection vis-à-vis third parties.

This language leaves much to interpretation. Even so, it clearly encourages courts and parties to take technology into consideration when weighing discovery burdens and costs. Implicit in this, it seems fair to say, is the court’s recognition that search, sampling and tools such as predictive coding can significantly reduce both the burden and cost of e-discovery.

With these new rules, Pennsylvania’s Supreme Court has made clear its intent to chart its own route on e-discovery, independent of federal jurisprudence. It will be interesting to see how this course develops. Even so, in their own way, these new rules add to the growing body of law that recognizes the increasingly essential link between sophisticated technology and cost-effective e-discovery.

Filed Under: Court rules, Predictive Ranking, Sampling, Search Tagged With: , , , ,
« Older Posts