Preface
The thoughts reflected in this piece stem from over 11,000+ hours of Japanese-language E-Discovery related projects based in Tokyo, Los Angeles, Phoenix, New York and Washington, DC, the majority of which as a Senior Project Manager for The CJK Group. As a lawyer licensed to practice in the United States, I’m also culturally and linguistically fluent in English and Japanese. I work directly with legal counsel, litigation support teams, paralegals and various E-Discovery software systems, including machine translation software. Most importantly I am deeply embedded within the multi-language review teams that carefully sift through the voluminous records that make up the corpus of electronic evidence that assist counsel in developing their factual understanding of their clients’ case posture. This article is Part II of a two-part series. (Part I can be found here)
When moving forward with a bilingual eDiscovery review, the most important role to be filled is the project manager position. A project manager can be anything from a detached HR representative overseeing 3-4 review projects at once with minimal day to day role with the substance of a review, to a fully integrated leader of a review team that makes the day to day decision-making, leadership, and quality control of a review. In bilingual eDiscovery reviews, due to the fact the managing law firm is unable to oversee the progress of the review or take an active role in its management, integrated hands-on project managers tend to be highly necessary in bilingual reviews.
However, what qualifications should an attorney look for when selecting a foreign language eDiscovery vendor’s Project Manager? What responsibilities will this project manager undertake? And, perhaps to clarify, what does a project manager actually do?
Management Types
First, let’s discuss the basic division of management types. Broadly speaking, eDiscovery vendors generally diverge in their hiring chiefly with regards to two main qualifications: bilingual vs non-bilingual managers; or attorney/non-attorney managers. A bilingual manager speaks the foreign language that is the subject of the foreign language eDiscovery review, while a non-bilingual manager speaks only English. An attorney manager is barred in a US jurisdiction, a non-attorney manager is not.
Therefore, within the bilingual eDiscovery industry, managers can be split into four different four models:
- Not bilingual; non-attorney
- Bilingual non-attorney
- Not bilingual; attorney
- Bilingual attorney
It is crucial, for reasons that will become clear below, to keep these four categories in mind when discussing pricing estimates with vendors and/or when determining the qualifications needed for a specific foreign language eDiscovery project. Consider for a moment, a vendor that quotes an hourly rate which is 1/3 that of a competitor, but whose quote is for an English-only non-attorney project manager, while the other vendor’s quote is for a bilingual attorney. What are the tradeoffs being made and how will these tradeoffs impact the trajectory of the review?
Goals of the Project Manager
To better understand the potential tradeoffs, we must first consider the goals of the project manager.
1) To conduct an accurate and uniform review.
A project manager must answer questions from reviewers regarding interpretations of the protocol, decide when to elevate questions to the managing associate and help reviewers apply importance consistently, particularly with regard to key and hot document designations. They must craft QC searches based on identifiable patterns, for example searches targeting issues of responsiveness that may be easily overlooked or misunderstood by reviewers.
2) To conduct an efficient review.
A project manager must ensure that the reviewers are working, and not being distracted or unfocused, identify “bottlenecks” in review which may cause excessive slowness and relatedly recommend changes to alleviate these issues. They must interface with technical vendors in implementing TAR, providing feedback and improvements.
3) Communicate progress and findings.
A project manager must be able to characterize findings and recommend areas of further investigation. They must assist counsel in characterizing findings and guide the managing associate to the most legally significant findings in a timely and succinct manner.
General Qualifications
Thus, to ensure the uniformity and accuracy in applying a law firm’s review protocol, what qualifications are therefore needed for the Project Manager of a bilingual document review? There are primarily two things the Project Manager must do: (1) lead the Review Team by answering questions while interfacing with the law firm to obtain clarifications; and, (2) craft customized search patterns to efficiently QC the dataset.
Search-based QC is a large topic that cannot be covered sufficiently here and will be explored more deeply in a future article. On a side note, there are some in the industry that believe keyword search is limited. While we understand this on the conceptual level, the nature of both the adversarial process coupled with a non-english dataset, the role of keywords must not be overlooked. Good Project Managers will identify patterns in documents that tip them off as to potential higher risks for inaccurate coding, especially in important documents, and perhaps help cull the data set, potentially saving time and money. This can only be done by a project manager with strong language skills, cultural knowledge, and legal proficiency. However, in our article today, we will focus upon how bilingual language skills and attorney training are necessary for leadership and clarity to effectively manage a bilingual review team.
Dilemma
Consider the dilemma faced by a project manager when there is a specific ambiguity in the protocol that governs how the review team analyzes foreign language documents for responsiveness. How would an ideal project manager deal with this? What skills will be necessary for the project manager to be effective in their role?
To start, let’s consider why one should care about uniformity or consistency. When there is an ambiguity or inconsistency in the protocol instructions, left unaddressed, the review team will be inconsistent in its application of the instructions. Such inconsistencies can dramatically increase the risk of accidental production of unresponsive or privileged documents or the production of undiscovered Hot documents. It could alternatively result in widely inconsistent and seemingly random production of documents that lead to discovery sanctions. Moreover, key pieces of factual knowledge may go unnoticed or even additional documents might unnecessarily be reviewed for months.
For our example, imagine the below instruction in a document review protocol. While not taken from any specific case, in my extensive experience, instructions with wording similar to this one is typical in any number of different kinds of litigation (FCPA, Antitrust, SEC investigations, etc.):
Many litigation attorneys may scratch their heads and say, this seems straightforward enough, why would this be a good example?
Meanwhile, many bilingual document reviewers will focus in on the five words: “or other accounting related documents.” When reading instructions on responsiveness and issue tags, competent reviewers are reading with one primary thing in mind—how broadly applied is this tag?
In other words, most reviewers are relatively uninterested in documents which clearly fit within the scope of the tag (annual financial statements for the company, etc.), and are instead concerned about non-obvious documents which might be included or excluded under this criteria—documents which they will inevitably find.
For example,
- Are all emails between accounting department personnel “accounting related?” All substantive emails? All financially related accounting emails?
- Are emails or documents involving the profitability of individual products considered “accounting related”? Or only those that show the financial state of the company as a whole? What about documents involving individual subdivisions or departments?
- Does “accounting related” include publicly available advertising materials that tout the total sales of products in certain countries (“best selling widget in Brazil with over 700,000 widgets sold!”)?
To be clear, some of the answers to these questions may be obvious to any litigator handling the case. In fact, that very assumption (that these judgments are obvious) is what often leads to these inadvertent gray areas within the protocol, which will subsequently require clarification for the reviewers.
That obviousness to the litigator will not be apparent to many reviewers because reviewers are not experts in that field of law. As generalists, most attorney reviewers will not be familiar with the ins and outs of the important areas of significance in those fields of law, or what kinds of evidence are expected. In some cases, the reviewer may not even be an attorney, but a bilingual paralegal (in certain languages due to the sheer lack of licensed attorneys in that particular language).
The difficulty in assuring the review is conducted in the way the managing law firm intends is dramatically increased when conducting reviews using bilingual non-attorney personnel, particularly review teams in foreign countries. I have considerable experience managing and working with bilingual paralegals in Tokyo. For most such personnel, the English language abilities of the paralegals was conversational, but that did not mean they could easily follow instructions on legal topics in English—meaning that considerable care had to be taken to provide extensive instruction and additional training in Japanese. I would also state that even if the team were all fully admitted Japanese attorneys, the issues highlighted above still remains. Full english language fluency and US-type thinking will not be present, thus requiring significant reliance on a strong Project Manager to balance out the difference.
In any case, how each reviewer internally decides to address ambiguities in the protocol may result in wildly different types of documents being included or excluded by this criterion. One reviewer, “Mr. Inclusive,” may include every last email in the accounting department, down to plans for accountants to go golfing, or every mention of the price or cost of any individual product. Another reviewer, “Ms. Exclusive,” may go in the radical opposite direction, and include only the annual financial statements or official accounting documentation. She may exclude documents discussing the figures to be used to calculate the annual financial statements within the accounting department.
Suggestions from the law firm along the line of “be more inclusive” or “cast a wide net” are usually insufficient to resolve this. Reviewers may question whether the intent of the managing law firm was to capture minutia such as brief advertising statements or individual product references to expected sales and costs, and include or exclude such documents. Asking reviewers to individually draw their own line in the sand on such an issue will not result in anything approaching uniformity—and often will fail to reflect the managing attorney’s intentions.
What is to be done?
If a phrase as seemingly innocuous as “other accounting related documents” can open up various questions, what can law firms do to ensure that their intentions are accurately and uniformly reflected in the final production?
This type of issue requires the leadership of a Project Manager.
The Leadership of the Project Manager: Good to Great
In practical terms, the first week or two of any review should involve the Project Manager heavily interacting with their team to identify and resolve as many issues of undetected “gray areas” like the one described above. A good Project Manager will engage with his or her team, encouraging reviewers to ask questions, keeping their ears open to discussions by the team on their confusion or uncertainty. A GREAT Project Manager anticipates likely ambiguities and questions, training the team to know which instructions contain ambiguities before they look at a document, so they can ask questions accordingly. Often a great project manager can easily function in both English and (other languages) to provide deep analysis and examples to ensure nothing is “lost in translation.”
If a reviewer inquires to the Project Manager, for example, whether an email within the accounting department, discussing (for example) what sales numbers or what accounting methodology to use with respect to a department, qualifies under the Financial Documents tag, a great Project Manager will instantly see the gray areas this instruction opened up under the “or other related documents.” Or even better, he or she may have anticipated this problem having read the protocol, and asked the team to be on the lookout for any ambiguity actually reflected in the documents.
This great Project Manager will now assess the seriousness of the ambiguity on several holistic criteria:
1) First, based on the descriptions of documents described in the protocol and supplemental documentation, is it overall obvious or ambiguous as to how the law firm would want this issue resolved?
2) Second, how important is the designation according to the law firm’s instructions? For example, if the documents would almost always be already responsive under other criteria, and the law firm stressed in training that the primary consideration is identifying Hot documents and consistent responsiveness calls, but the specific issue tags are more of a guideline, perhaps the distinction is not as important.
Conversely, if the resolution of the ambiguity would change the responsiveness of a large swath of documents, or impact a determination of whether a document should be considered Hot (i.e. documents likely to be highly significant to key substantive issues within the case), the document and the resolution of this ambiguity is more likely to be important.
3) Third, how common are documents that fall into this gray area? If after reviewing 10,000 documents, the issue comes up in only one or two documents, perhaps it is not a serious issue. If thousands of documents decisions hinge upon this criteria, it would appear to be more serious.
In going through this assessment, a great Project Manager is ultimately trying to determine whether the issue is worth raising to the managing associate as a question or request for clarification from the review team. If documents at issue are extremely rare and unrelated to “Hot” issues in the review, the ambiguity is unlikely to be of high importance. This same Manager may consequently make a decision based on the totality of circumstances and include a minor clarification in a team-wide email at a later time, but not waste the team or managing associate’s time addressing such a minor issue. On the other hand, if the issue is frequent or, for example, rare but related to issues of central importance in the case, the Manager will craft a question to the managing associate to resolve the ambiguity.
You may be asking, why would such a criterion be necessary? I don’t want my project manager screening any questions, I want my associate to be forwarded every single question and ambiguity that comes up in the case.
Unfortunately, in most cases, the sheer volume of questions from reviewers can be overwhelming to a managing associate, who likely has many competing bids for their time. A managing associate may, due to the resulting time constraints, be forced to begin answering many questions with a simple “use your best judgment”—an answer which essentially abandons any sense of uniformity, permitting Mr. Inclusive and Ms. Exclusive to continue coding documents in radically different ways. Even worse, if half a 30-person team codes like Mr. Inclusive, and half like Ms. Exclusive, the entire data set may look entirely haphazard and randomly produced if audited.
The Rub: It’s all Greek to me!
These issues tend to be naturally compounded in bilingual reviews. Unless the managing associate is also bilingual, the associate is unable to read examples of documents. Even if the Associate could read the documents in the native language, the sheer volume of ESI will negate their ability to practically function as a Project Manager. They are merits or discovery counsel, not review managers. Without a Bilingual Project Manager, case teams are forced to rely upon the descriptions of documents from the bilingual reviewers themselves—a path which often leads to consistency problems. Often, the reviewer will already have an answer in mind. Unconsciously, he or she will likely describe the document in a way that is intended to draw the desired answer. If two reviewers provide two different sets of description for a similar document, the associate can easily provide opposite answers without even realizing he or she has done so—further confusing the reviewers.
Likewise, the non-bilingual Project Manager cannot read the documents relating to the questions they receive, leaving them with two options.
- The Manager simply relies upon the descriptions provided by the reviewer; or,
- Find a trusted bilingual reviewer to rely upon to provide guidance in understanding the documents and answer the questions according to that reviewer’s guidance
The problems with Option (1) are as discussed above—descriptions of documents may be selectively chosen to advance a reviewer’s argument for why he or she already thinks it should, or should not be Responsive/Issue Tagged etc., consequently, relying on those descriptions can result in confusing and contradictory instructions to the team.
The “Team Lead Model,” Option (2), finding a trusted bilingual reviewer to assist, comes with a different set of problem and deficiencies. Either the non-bilingual manager becomes an unnecessary and unhelpful rubber-stamper, or the non-bilingual manager chooses a poor Team Lead with disastrous results due to his or her inability to review and evaluate the work of the Review Team. To put it mildly, the English speaking project manager does not have the capacity to evaluate who is accurate in their analysis conducting foreign language review. It’s a guessing game that could go well or wildly off course.
Across the bilingual eDiscovery industry, the term “Team Lead” usually refers to a bilingual reviewer who is given the informal title of “Team Lead.” The Team Lead works to assist an English-only Project Manager and, ideally, make up for all the linguistic deficiencies of the English-only Project Manager. They do so by impartially describing all of the documents, relying on the superior judgment of the Manager who is able to then provide the analysis for which they were specifically hired.
In practical terms, what often follows is one of two broad problematic possibilities:
Are you a 5th Wheel Project Manager: Let’s get into the weeds:
First, when the team has a competent Team Lead, you can have the “fifth-wheel Project Manager” situation. As the only person who can actually look at and comprehend the documentation is the Team Lead, the team primarily interacts with the Team Lead, asking questions and seeking clarification. The Team Lead obtains verification from the PM for their decision-making—but since the English-only Project Manager cannot actually read the contents of the documents, and word for word translation is too time consuming, the Project Manager can usually do nothing but rubber stamp most decisions made by the Team Lead.
After all, if the Project Manager cannot read the document, the situations where they feel comfortable overruling the Team Lead’s recommendation will be ordinarily few and far between. Alternatively, but much worse, the Project Manager could respond by arbitrarily overruling the Team Lead despite his or her lack of comprehension.
Where there is a competent Team Lead, the team will look at the Team Lead as the actual substantive decision-maker and leader. Meanwhile, the Project Manager simply will be a detached advisor—an HR rep who parrots to the law firm’s managing associate the real decisions being made by the Team Lead—hence, the nickname the “Fifth Wheel Manager.”
One other far worse alternative is when an incompetent Team Lead is chosen by the Project Manager.
The problem again circles back to the inability of the English-only Project Manager to look at the documents and determine the reasonableness of the Team Lead’s recommendations and analysis of those documents. If what the Team Lead says sounds reasonable and confident, the Project Manager has no personal way to truly verify this analysis on their own. If too much trust is placed in an individual, or a small circle of incompetent reviewers, this can have disastrous consequences that can go undetected until the problem is extraordinarily expensive to fix.
This scenario is not just a hypothetical. It has been a semi-regular occurrence throughout my career to have engagements where the primary purpose was a request to fix a review that previously went off the rails at a different company. Re-reviewing portions of the data, a second pass through privilege documents, etc.
Messy Reviews & Wasting Clients Money
Years ago, while working for a previous employer, I also personally worked on a similar, lengthy document review as a first-level reviewer. The project manager spoke only English, and furthermore was secluded in the manager’s office. The Manager barely interacted with the first-level reviewers. Everything ran through the Team Lead, whom, many of the reviewers became convinced over time, both had tremendously poor legal judgment and insufficient Japanese language skills. Nonetheless, the Project Manager spoke only English, so the Manager had no idea how haphazard the decision-making was—and the ordinary reviewers were afraid of speaking up against a Team Lead who seemed to have the complete trust of the Manager, in fear of retaliation.
The problem was not discovered until very late into the project, after an enormous amount of time had been spent and translations from a 3rd party translation company began coming back. The supposedly important documents completely differed from the descriptions provided by the Team Lead to the Project Manager and to the client law firm. Further investigations into the accuracy of the review also revealed major inconsistencies and errors owing to the Team Lead’s decision-making.
The Team Lead was fired, a different one was appointed. But by this point, much damage was already done. At tremendous expense, numerous documents had to be reevaluated under newly clarified criteria.
Final Thoughts: “Team Lead” model:
When relying upon the Team Lead model, in many ways, the Team Lead is far more important to the success or failure of the Review than is the English-only Manager. Thus, the Team Lead’s position should be scrutinized even more—yet in many cases, the position is strictly informal, and often does not even include a pay raise over that of the ordinary reviewer (merely an unspoken promise of being prioritized for being offered work more frequently by that service provider).
Which would lead to the question—why would you devote extra resources to a Project Manager position, and not to hiring the best Team Lead you could? Why should you not make the Team Lead the manager in the first place?
The Value of a Legal Mind
Revisiting our basic division of management types, one of the major splits is between attorney Project Manager versus non-attorney Project Managers. Bilingual eDiscovery reviews often include non-attorney paralegals due to the realities of scarcity of labor in certain languages, such as Japanese, Bahasa, German, etc. The managing associate needs to determine whether the Project Manager needs to be an attorney, and there are many good reasons that they should be.
First, if a review team is made up of attorneys, or a mixture of attorneys and paralegals, this presents an extreme leadership challenge for a non-attorney manager. Simply put, many attorneys will be dismissive of the clarifications or interpretations of the law firm’s instructions when provided by a non-attorney. This often will not manifest itself in outright insubordination (although it may).
However, in most cases, it will be reflected in a significant reduction in the amount and types of questions reviewers bring to the manager. Where a question involves any kind of subjective call, many attorneys will simply assume that even if they are not sure, surely their own legal judgment is more likely to be correct than that of a non-attorney. The question, therefore, goes unasked.
As noted earlier, questions from reviewers can be one of the most important and instructive sources of information for a Project Manager on where ambiguities and potential points of inconsistent review exist within the protocol and across the data set. Qualifications that suppress free question-asking put non-attorney project managers at a major disadvantage.
If a team, on the other hand, is composed only of bilingual paralegals, attorney leadership for the team becomes even more crucial. Due to the lack of an attorney on-site, the project manager will often be looked towards for guidance of legal language and interpretation. They will often find themselves in a position where they must explain legal concepts, issues or significance to the paralegals in answering questions. For reasons that should be obvious, attorneys will be better equipped to handle such a situation than a non-attorney.
While a non-attorney Project Manager cannot provide guidance and answer questions involving legal interpretation, their job is often relegated to facilitate reviewer questions. In assessing the significance and importance of whether a question should be elevated, however, legal knowledge can be crucial. An attorney project manager speaks the language of litigation, or at least is trained in the basics of legal analysis. Thus, in reading the protocol or in discussion with the associates, they are better equipped to have a greater holistic understanding of what types of evidence is desired, feared, or particularly necessary.
A non-attorney project manager could attempt to forward to the associate questions every time such an issue potentially comes up—but this type of approach often results in exceptionally time-consuming engagements on the part of the managing associate. In my experience, many such project managers simply attempt to use their own judgment to do as best they can. But without formal legal training, getting non-attorney managers on the same wavelength as the law firm, particularly in a case of legal complexity or subtlety, can be a significant and time-consuming challenge for the managing law firm.
For the reasons discussed above, to ensure the uniformity and accuracy of the review in leading a bilingual document review, literacy in the language and being an attorney will be important requirements for a good Project Manager in many cases. In some cases, the fact finding nature of the review is so simple that uniformity and accuracy are minimal concerns—for example, a review in which the firm is producing virtually every document hitting upon search terms and has already admitted fault, and the reviewers are only looking to double check against false hits and technical errors.
In Sum
But for a review where accuracy and uniformity are concerns and where complex fact patterns exist, the use of a Project Manager that is lacking in either bilingual or attorney qualifications will lead to a firm necessarily accepting significant risk exposure to possibilities of inconsistent application of instructions. That is, the increased risk of accidental production of unresponsive or privileged documents, the missing of hot and/or key documents, or widely inconsistent and seemingly random production of documents that might lead to discovery sanctions.
More often than not, relying upon project managers with bilingual and attorney expertise are cost efficient forms of risk management. So next time you are faced with a cross-border matter, be sure to ask your litigation support teams and/or the e-discovery practice groups how they actually handle non-english reviews. Having been in the industry for nearly a decade exclusively managing foreign language assignments, it is quite shocking how many vendors overlook the importance in culture, diversity & bilingualism. While this might often seems administrative, this can greatly impact how one renders effective counsel. Although we do not provide any advice on substantive or legal matters, we at The CJK Group know a thing or two about effective CJK language document reviews!
