8. A Forensic Perspective: Testing and Verification of Assertions
An independent monitor commonly assembles a team of professionals to assist in the discharge of his or her duties and obligations under the settlement agreement. Forensic professionals are crucial members of that team as they provide technical and industry expertise in the financial, computational and technological aspects of the settlement agreement. Most commonly, as discussed more fully in Chapter 8, an independent monitor retains the services of forensic professionals to assist with various aspects of his or her role in overseeing the company’s compliance with the settlement provisions, including, for example, the verification of the eligible and affected settlement agreement populations relevant to the issues or wrongdoing that led to the settlement, as well as the application and provision of remediation to those within the affected population. In short, the forensic professionals’ role is to provide vital support in the technical areas required to fulfil the independent monitor’s mandate under the settlement agreement.
As discussed in more detail elsewhere this guide, though a settlement agreement will describe the independent monitor’s mandate and provide a general framework, typically a work plan is drafted that establishes the specific processes, procedures and methodologies to validate and confirm that the settling party has satisfied its obligations under the settlement agreement, including the population parameters, sampling methodology, if applicable, and the testing framework. The work plan typically addresses the following topics:
- a description of the parties to the settlement agreement, including their authority, roles, responsibilities and requirements;
- a description of the team employed by the independent monitor to validate the eligibility of the settling party’s activities and the satisfaction of the terms of the settlement agreement;
- a discussion of the review techniques to be used by the independent monitor’s forensic professionals, including an appropriate sampling process and testing methodology;
- a description of certain tests, often called metrics, by which to identify the accuracy and completeness of the eligible and affected population identified within the settlement agreement and ensuring that those metrics are rigorous and commensurate with the size, complexity and risks associated with the same;
- the protocols and metric testing templates under which the testing criteria, procedures and assessment are conducted and the parameters of testing for each action or item within the independent monitor’s mandate to measure whether a given action or item in a population ‘passes’ or ‘fails’ the metric test; and
- the reporting processes of the independent monitor.
Forensic professionals’ role
Among other things, independent monitors are appointed to assist the courts or regulatory bodies in enforcing rulings or ensuring compliance with regulations. Often referred to as ‘consumer relief’, settlements may include remediation of financial injury caused to a student, patient, borrower or customer – essentially any potentially harmed party as a result of the issue at hand or wrongdoing. Regardless of the type of settlement or court order, the independent monitor’s forensic professionals will carry out a separate validation and confirmation of the settlement agreement’s affected population (or populations) and determine whether the compliance or remediation claimed by the settling party complies with the terms of the settlement agreement. In performing this work, the forensic professionals apply the methodology and structure outlined in the work plan and agreed metric testing templates by accessing the settling party’s systems of record (SORs) relating to its operations and reviewing the various inputs required to undertake the settlement agreement population determination, and for settlements requiring consumer relief, remediation calculations for each action or item. Specifically, the forensic professionals conduct an independent examination to determine whether each of the actions or items identified within the settlement agreement population are, in fact, accurate, eligible and properly remediated, which may include a financial analysis and, if applicable, calculation to determine the correct amount of consumer relief required by the settlement agreement.
As noted, when preparing the work plan, a testing methodology is agreed and established for validating both the affected population and the remediation of the identified actions or items subject to the behaviour or actions leading to the settlement. Based on this testing methodology, the forensic professionals will first develop templates under which testing and validation will be performed, and establish protocols for identifying and evaluating the settlement agreement populations. The methodology within the work plan and respective testing templates specifically identifies the process, procedures and methods by which the forensic professionals will perform their work and assess both the applicability of the population identified and the remediation claimed pursuant to the settlement agreement’s requirements. The testing templates, developed primarily by the forensic professionals, provide step-by-step instructions for carrying out the testing and validation to test compliance with the settlement provisions.
The forensic professionals then proceed to independently test and verify whether the identified population and the claimed remediation comply with the terms of the settlement agreement. Typically, the result of that process is a report through which the forensic professionals confirm to the independent monitor that the proposed populations and remediation meet the requirements, conditions and limitations set forth in the settlement agreement. In addition, when applicable, the forensic professionals will indicate that the settling party has fully satisfied its settlement agreement obligations.
Systems of record
The forensic professionals’ review is conducted by accessing and reviewing various data and documentation within the settling party’s SORs, which often includes its vendors’ SORs. In turn, access to the settling party’s work papers and supporting documentation, including confirmation of the settlement agreement population identification process through which the work papers are identified, is required and must be sufficient for the forensic professionals to substantiate and evidence the accuracy and validity of the work performed. This process includes capturing and saving within the work papers images from the settling party’s SORs, evidencing the relevant data and performing a review of any queries used to derive the population.
As the SORs are often heavily relied upon to conduct the necessary validation, forensic professionals are required to perform a quality review of the SORs before relying on them. Because the underlying information and processes are highly confidential and proprietary in nature, this presents a significant challenge for the parties to the settlement agreement. In turn, it is not uncommon for forensic professionals to rely on internal testing (e.g., internal reports, quality control testing) and external testing (e.g., reports on the organisation’s controls, regulatory reviews). Although often complicated and not without a fair amount of negotiation, it is a necessary step to confirm the validity and accuracy of the compliance process. This issue is often exacerbated when dealing with third-party vendors, who are not direct parties to the settlement agreement. In that situation, the settling party relies upon the data, but typically does not have direct access to the SORs.
Identification of the full affected population
The independent monitor’s mandate may also include certifying the accuracy and completeness of the identified affected population within the settlement agreement. In so doing, the independent monitor typically engages forensic professionals to understand how the settling party determined the relevant population and to validate the accuracy and completeness of the identification of actions or items within the settlement agreement.
In some cases, the settling party agrees to provide the independent monitor’s forensic professionals with access to the relevant documentation and SORs that are necessary to enable the forensic professionals to verify the settlement agreement population. The SORs may include queries or coding and other information reasonably expected to be relied upon, to enable the forensic professionals to understand the evidence that is needed to assess whether the settling party has accurately identified the population within the settlement agreement. In those instances, however, it may require the forensic professionals to extract the supporting documentation (active and archived) from the settling party’s SORs. This approach includes the forensic professionals’ development of independent queries within platforms and the use of independent access to the source systems to extract the populations, as well as manual validation and a comparison of the information acquired to that identified within the settlement agreement.
Alternatively, the independent monitor’s forensic professionals may only be required to review the method by which the settling party determined the number of actions or items approximated in the settlement agreement. This means that the forensic professionals only assess the means and mechanisms used by the settling party as part of the process to identify the population subject to the settlement agreement. Specifically, the forensic professionals will evaluate whether the settling party’s population identification work papers are substantially accurate and independently assess whether the results of the population analysis effectively substantiate and confirm the accuracy and validity of the population identified within the settlement agreement population. In such a case, the settlement agreement would require the settling party to provide the independent monitor’s forensic professionals with documents and information relevant to the settlement agreement that are reasonably necessary to complete the independent monitor’s mandate.
In any event, a constant consideration and focus of all parties to the settlement agreement is the need to provide the necessary information to support testing under the work plan and metric testing template in a secure platform that protects the integrity and confidentiality of the information, yet still provides sufficient visibility and detail to accurately validate completeness and compliance with the settlement agreement requirements. The settling party’s records and related systems, which reflect its operations, are typically maintained within internal and external platforms. As such, the forensic professionals will typically hold meetings with representatives of the settling party and its vendors on multiple occasions to obtain an understanding of the functions, processes, applications and proposed approaches for testing and validation, including the retrieval process for documentation across enterprise systems and archives.
Certain settlement agreements specifically identify metrics by which to assess compliance of the settlement agreement effectively. These metrics address distinct matters within the settlement agreement that may have unique complexity or specific time frames (or both) or be associated with a specific population within the settlement agreement. Metrics are contemplated and designed to specifically evaluate whether the settling party is fulfilling its distinct, identifiable obligations under the settlement agreement for these individual, unique matters. The metrics are mapped to the terms of the settlement agreement and are implemented via the standards established within the work plan and respective metric testing template.
The population and testing methodologies for each metric are generally set forth in the work plan, which includes the documentation required for validation, and the development of detailed metric testing templates for the independent monitor’s forensic professionals to use in reviewing work papers in connection with confirmation of the same. These metric testing templates set forth the specific rules for how different actions or items are identified, the eligibility of these actions or items and how remediation will be determined.
Identification of the metric populations
To select the relevant population for each metric, the settling party will need to evidence the mechanisms by which the identified metric population is extracted and identified from within the full settlement agreement population or from within the settling party’s systems, including its vendors’ SORs, if applicable. These queries are performed by the independent monitor’s forensic professionals, whose review and evaluation also include the preparation of documentation which evidences that the respective metric populations conformed in all substantial respects to the settlement agreement.
In those matters in which the metric populations are extracted and identified from within a consistent and verifiable settlement agreement population, it is not uncommon for the settling party to create a controlled environment for population identification, by which metric level population testing and validation can be conducted without affecting the settling party’s ‘live’ SORs. In those cases, it is imperative to validate the integrity of the data and the accuracy of the full settlement agreement population prior to applying additional rules associated with metric population identification and validation. Typically, the settling party’s internal infrastructure and technology team will construct a controlled population environment for identifying the populations that are subject to manual item level testing by the independent monitor’s forensic professionals to ensure accuracy and completeness of each metric population.
The forensic professionals will then identify and extract metric populations for testing from the controlled population environment. This process includes the development of queries within the controlled population environment to independently extract metric populations, and to carry out further data analysis and comparisons to the entire settlement agreement population to ensure that all actions or items, as well as fields necessary for validation, were accurately captured. This analysis includes the forensic professionals investigating actions or items that were not included in the metric population but have been identified as meeting the requirements for inclusion.
The forensic professionals document their analysis and the logic used to identify the metric populations from the settlement agreement population. In addition, it is not uncommon to apply a threshold error rate to determine whether the settling party’s assessment with respect to each metric population is accurate. Using this approach, if the metric population, as identified by the forensic professionals, was greater than the estimated metric population, as provided by the settling party, by an agreed percentage, as identified within the work plan or the metric testing template, the settling party would ‘fail’ with regard to the identification of the applicable metric population, and the appropriate course of action, as identified in the work plan, would be required.
In certain circumstances, the independent monitor’s forensic professionals’ approach to evaluate the settling party’s compliance with the metrics eligibility testing and determination of remediation is performed on a sample basis, rather than a review of every action or item in the population. In such cases, the forensic professionals use statistical parameters and a sampling methodology explicitly delineated within the work plan or respective metric testing template. Based on these statistical parameters and sampling methodology, the forensic professionals select a sample of actions or items from the applicable population. The samples selected by the forensic professionals from the appropriate populations will be based upon a detailed review of the settling party’s relevant records and include a number actions or items that are statistically significant. As such, the forensic professionals prepare detailed work papers reflecting its population and sampling review and analysis, which includes verification of the sampling tool used by the forensic professionals and validation of other relevant sampling methodologies.
Sampling is a scientifically valid method of using a subset (sample) of a group (population) about which one wishes to draw inferences to estimate some property or characteristic of that population without having to examine each and every item of the population. It involves drawing a random sample from a population in which each element item within the population has an equal probability of being selected and included in the sample. The randomness of the sample allows one to extrapolate the sample results to the population because it allows a statistician to make the assumption that the relationships in the sample are like the relationships in the population.
Extrapolating from the sample to the population involves measures of reliability and precision, known as ‘confidence levels’ and ‘margins of error’. The confidence level is the probability that the population value will fall within a specified range around the value extrapolated from the sample. The margin of error is synonymous with sampling error and is most commonly referenced in political polls. As a simple example, assume that a population of 35,000 actions are identified within the settlement agreement as eligible and entitled to receive remediation. Rather than testing each of the 35,000 actions in the settlement agreement population, a completely random sampling methodology with a 95 per cent confidence level and a margin of error of plus or minus 5 per cent indicates that a randomised sample size of 380 actions would enable a tester or reviewer to obtain an accurate, reliable representation as to the settlement agreement population as a whole.
The independent monitor’s forensic professionals will reach their conclusions by randomly selecting statistically valid samples from actions or items that the settling party deemed eligible and sought remediation pursuant to the settlement agreement. Each of the metric samples is drawn from a separate and distinct metric category population identified within the work plan and metric testing template, and each is treated as a testing population. These random samples are typically selected by using an established, licensed analysis software product (e.g., Microsoft SQL Server Management Studio or Microsoft Excel). In determining the sample size, the forensic professionals, in accordance with the work plan and metric testing template, use the ‘confidence level’ and ‘margin of error’ methodology delineated in the work plan and metric testing template, and as described in the previous section. Through this process, the forensic professionals identify a statistically significant number of actions or items for testing and review.
At that point, the metric population is randomised and each action or item is assigned a computer-generated random number. Using the sample size calculator, a certain number of random actions or items are selected for testing. These are then typically loaded into a database from which the forensic professionals’ testing is performed. The forensic professionals’ work papers will include screen shots of each step to evidence that the sampling methodologies were carried out properly.
For each action or item, the independent monitor’s forensic professionals determine whether it met the criteria described within the settlement agreement and based on the assembled data for that action or item. For any action or item determined not to meet the criteria of the settlement agreement, the forensic professionals would conclude that the affected population was not successfully remediated or did not meet the compliance requirement. For each action or item determined to meet the criteria for remediation, the forensic professionals ensure that the settling party had correctly reported the remediation completed by verifying key data points. In many work plans and metric definition templates, a tolerance level for error is also established. This serves to define what constitutes a ‘fail’ on a single action or item based on a pre-determined threshold (typically a count, an amount or a percentage) beyond which the variance between the outcome determined by the independent monitor’s forensic professionals and the outcome reported by the settling party is reportable as a ‘fail’.
In addition to verifying whether the remediation met the applicable criteria for the actions or items within the settlement agreement population, for settlements that include consumer relief, the forensic professionals often determine the amount of remediation within the population tested (actual remediation amount) and compare that with the amount of remediation claimed by the settling party for the same actions or items (reported remediation amount). If the amounts are equal, or if the actual remediation amount is within a certain percentage of the reported remediation amount, as identified within the work plan or metric testing template, the reported remediation amount will be deemed correct and confirmed by the forensic professionals. If, however, the forensic professionals determine that the reported remediation amount for the population exceeds the actual remediation amount by more than the percentage established within the work plan or metric testing template, that population would ‘fail’ the forensic professionals’ review and this failure would be communicated to the independent monitor.
When there is a ‘fail’, the settling party is required to implement corrective measures on the failing population, as determined in the work plan (discussed further in the section titled ‘Non-compliance’, below). Generally, however, when forensic professionals determine that the settling party has performed remediation on a larger population than was stipulated (or greater consumer relief than required) in the settlement, independent monitors accept the remediation as reported.
Ultimately, the forensic professionals confirm the eligibility of the actions or items identified by the settling party and whether the remediation claimed is accurate and conforms to the requirements of the work plan and metric testing template. By so doing, the independent monitor can determine whether the settling party is in compliance with the settlement agreement.
If the independent monitor’s forensic professionals conclude that the settling party has not yet met the settlement agreement’s standard or fulfilled the conditions required under the assessment, the independent monitor will be informed and will send the settling party a report of the assessment. Typically, the settling party can provide additional information to the independent monitor for consideration and can object to the assessment.
If, however, even with the additional information provided by the settling party, the independent monitor and his or her forensic professionals still believe the settling party has not yet met the settlement agreement’s standard or fulfilled the conditions required by the settlement agreement, the settling party will submit a corrective action plan to the independent monitor, for his or her approval. The independent monitor’s forensic professionals will validate or confirm the root cause of the error and determine whether the plan is sufficient to correct the error going forward. Once the corrective action plan has been vetted and approved by the independent monitor, the forensic professionals will determine whether the settling party has fully and successfully implemented the corrective action plan and, if applicable, remediated any harm caused by the failure. Typically, metric testing will halt during this period of vetting and approval. Only once remediated and tested will the settling party again submit actions or items for review and the forensic professionals can resume testing.
The independent monitor will ultimately issue a report that describes the independent monitor’s forensic professionals’ role in establishing, assessing and overseeing the settlement agreement. This report will explain the forensic professionals’ role in validating the applicable populations, confirming that the eligible actions or items were correctly identified and whether the appropriate remediation was provided. In summary, the forensic professionals are a vital part of an independent monitor’s team and mandate to ensure that a settling party adheres to its obligations within the settlement agreement.
1 Loren Friedman is a senior manager, Thomas Cooper is a managing director and Nicole Sliger is a partner at BDO USA, LLP.
2 The independent monitor’s team, including the forensic professionals, are independent and free from relationships or conflicts with the parties to the settlement agreement, as such conflicts would undermine the public trust and confidence in the objectivity of the work performed under the settlement agreement. Even though the independent monitor’s team are typically paid by the settling party for the services performed, it is imperative that the settling party does not have the power to direct or control the independent monitor’s team’s work.
3 The work plan is typically developed through multiple meetings and consultations between the parties to the settlement agreement in which the unique details specific to the settling party are discussed and considered, such as the settling party’s organisational structure, operations, systems of record, roles and responsibilities of the personnel involved, and other information reasonably expected to be relied upon, among other applicable topics.
4 Note that the work plan does not limit the responsibilities or roles established under the settlement agreement, but rather supplements the settlement agreement and provides added definition and detail. It is not uncommon for the work plan to be amended from time to time, as necessary and as agreed by the applicable parties to the settlement agreement, including the independent monitor.
5 Depending on the terms of the settlement agreement, the settling party may also establish and make operational an independent quality control review group, which consists of employees organisationally separate and distinct from the settling party’s functions and operations or business line, responsible for complying with the settlement agreement. The independent monitor’s forensic professionals typically would conduct a review of the independent quality control review group programme, including the qualifications of the personnel assigned to the group and the internal controls designed to ensure independence from the settling party’s settlement agreement business or operational units.
6 As an example, under Metric 2 of the settlement between the United States Trustee Program and Wells Fargo, Bank, NA (Wells Fargo), the forensic professionals were ‘required to evaluate whether after March 31, 2015, Wells Fargo timely performed and communicated annual escrow analyses for debtors in Chapter 13 bankruptcy cases as required by the Real Estate Settlement Procedures Act and its implementing Regulation X’. The Metric 2 population was ‘selected on the basis of individual occurrences of annual escrow analyses for accounts in Chapter 13 Bankruptcy Cases that had a 12-month escrow analysis anniversary between April 1, 2015 and June 30, 2016 as projected on the immediately preceding annual escrow analysis performed by Wells Fargo’. See Final Report of Lucy Morris – Independent Reviewer Overseeing Compliance with Settlement Between the United States Trustee Program and Wells Fargo Bank, N.A. (30 March 2018), p. 17, at https://www.justice.gov/ust/file/wells_fargo_compliance_report.pdf/download.
7 Similar to the work plan, metric testing templates may be amended or augmented as needed during the review to assess compliance effectively.
8 Owing to the complexity and importance of this analysis, the independent monitor’s forensic professionals and settling party will commonly hold regular due diligence and population meetings to discuss issues relating to population identification and related topics. This is in addition to the typical regular scheduled meetings between the settling party and the independent monitor’s team.
9 Sampling is typically applied in situations in which a full population analysis is unduly difficult, impractical, voluminous, cumbersome or cost prohibitive.
10 ‘When a statistic is significant, it simply means that you are very sure that the statistic is reliable . . . Significance is a statistical term that tells how sure you are that a difference or relationship exists.’ – ‘Statistical Significance’, StatPac, at https://www.statpac.com/surveys/statistical-significance.htm
11 See, e.g., Stock, J Stevens and Hochstim, Joseph R, ‘Commercial Uses of Sampling’, Journal of the American Statistical Association 43, No. 244 (1948), at pp. 509 to 522; ‘Sampling/Opinion Surveys’, Manual for Complex Litigation, 4th ed. (Washington, DC: Federal Judicial Center, 2004), § 11.493; Deming, W Edwards, Sample Design in Business Research (New York: John Wiley & Sons, 1960); Levy, Paul S and Lemeshow, Stanley, Sampling of Populations: Methods and Applications, 4th ed. (Hoboken, NJ: John Wiley & Sons, 2008), p. 22.
12 See Cochran, William G, Sampling Techniques (New York: John Wiley and Sons, 1977), pp. 9 to 10; Thompson, Steven K, Sampling (John Wiley & Sons, 2012), p. 3.
13 See, e.g., Kish, Leslie, Survey Sampling (New York: John Wiley & Sons, 1995), pp. 14 to 15; see Beecher-Monas, Erica, Evaluating Scientific Evidence: An Interdisciplinary Framework for Intellectual Due Process 65 (2007); see also id. at 66 (‘The goal of the scientific standard – the 95 percent confidence interval – is to avoid claiming an effect when there is none (i.e., a false positive). Scientists using a 95 per cent confidence interval are making a prediction about the results being due to something other than chance.’). ‘[A h]igher confidence level requires a larger sample size’ – ‘Sample size calculator’, Raosoft, at www.raosoft.com/samplesize.html.
15 ‘Sample size calculator’ (footnote 14, above).
16 An additional number of actions or items may be selected in the event that replacements are needed.
17 Some settlement agreements not only require changing processes and procedures, but also contain a component in which potentially affected parties are provided relief (i.e., monetarily compensated) for previous actions of the settling party. In these types of settlements, the amount of monetary relief is defined in the settlement agreement.
18 A threshold error rate or percentage is a quantification of an acceptable and unacceptable testing error rate based on the action or item count or the remediation amount established under the work plan or the metric testing template which, when exceeded, is a ‘fail’ under the settlement agreement. This rate is specifically negotiated among the parties to the settlement agreement, typically when developing the work plan.
19 The forensic professionals also confirm that there are no duplicate actions or items within the population as the settling party can only receive remediation in relation to actions or items for which it has not previously submitted.