The Forensic Perspective: Testing and Verification of the Assertions Made Within Settlement Agreements

An independent monitor commonly assembles a team of professionals to assist in the discharge of his or her duties and obligations under the settlement agreement. Forensic professionals^[2] are crucial members of that team, who provide technical and industry expertise in the financial, computational and technological aspects of the settlement agreement. Most commonly, as discussed more fully in Chapter 13, an independent monitor retains the services of forensic professionals to assist with various aspects of his or her role in overseeing the company’s compliance with the settlement provisions. For example, this can include the verification of the eligible and affected settlement agreement populations relevant to the issues or wrongdoing that led to the settlement, as well as the application and provision of remediation to those within the affected population. In short, the forensic professionals’ role is to provide vital support in the technical areas required to fulfil the independent monitor’s mandate under the settlement agreement.

Work plan

Although a settlement agreement will describe the independent monitor’s mandate and provide a general framework, a work plan is commonly drafted that establishes the specific processes, procedures and methodologies to validate and confirm that the settling party has satisfied its obligations under the settlement agreement, including the population parameters, sampling methodology, if applicable, and the testing framework.^[3] The work plan^[4] typically addresses the following:

• a description of the parties to the settlement agreement, including their authority, roles, responsibilities and requirements;
• a description of the team^[5]employed by the independent monitor to validate the eligibility of the settling party’s activities and the satisfaction of the terms of the settlement agreement;
• a discussion of the review techniques to be used by the independent monitor’s forensic professionals, including an appropriate sampling process and testing methodology;
• a description of certain tests, often called metrics, by which to identify the accuracy and completeness of the eligible and affected population identified within the settlement agreement, ensuring that those metrics are rigorous and commensurate with the size, complexity and risks associated with the settling party’s compliance obligations;
• the protocols and metric testing templates under which the testing criteria, procedures and assessment are conducted and the parameters of testing for each action or item within the independent monitor’s mandate to measure whether a given action or item in a population ‘passes’ or ‘fails’ the metric test;
• the reporting processes of the independent monitor; and
• the protocols to follow in the event that the settling party and the independent monitor disagree on an issue.

Forensic professionals’ role

Among other things, independent monitors are appointed to assist the courts and regulatory bodies in enforcing rulings or ensuring compliance with regulations. Often referred to as consumer relief, settlements may include remediation of financial injury caused to a student, patient, borrower or customer – essentially any potentially harmed party as a result of the issues at hand or wrongdoing. Regardless of the type of settlement or court order, the independent monitor’s forensic professionals will carry out a separate validation and confirmation of the settlement agreement’s affected population (or populations) and determine whether the compliance or remediation claimed by the settling party complies with the terms of the settlement agreement. In performing this work, the forensic professionals apply the methodology and structure outlined in the work plan and agreed metric testing templates by accessing the settling party’s systems of record (SORs) relating to its operations and reviewing the various inputs required to undertake the settlement agreement population determination, and for settlements requiring consumer relief (i.e., monetary relief to the parties affected by the actions of the settling party), and the remediation calculations for each action or item. Specifically, the forensic professionals conduct an independent assessment to determine whether each of the actions or items identified within the settlement agreement population are, in fact, accurate, eligible and properly remediated. This may include a financial analysis and, if applicable, a calculation to determine the correct amount of consumer relief required by the terms of the settlement agreement.

Testing methodology

When preparing the work plan, a testing methodology is agreed and established for validating both the affected population and the remediation of the identified actions or items subject to the behaviour or actions leading to the settlement. Based on this testing methodology, the forensic professionals will first develop templates under which testing and validation will be performed, and establish protocols for identifying and evaluating the settlement agreement populations. The methodology within the work plan and respective testing templates specifically identifies the process, procedures and methods by which the forensic professionals will perform their work to assess both the applicability of the population identified and the remediation claimed pursuant to the terms of the settlement agreement. The testing templates, developed primarily by the forensic professionals in consultation with the independent monitor and taking into consideration the records of the settling party, provide step-by-step instructions for carrying out the testing and validation to test compliance with the settlement provisions.

The forensic professionals then proceed independently to test and verify whether the identified population and the claimed remediation comply with the terms of the settlement agreement. Typically, the result of that process is a report through which the forensic professionals confirm to the independent monitor that the proposed population and remediation meet the requirements, conditions and limitations set forth in the settlement agreement. In addition, when applicable, the forensic professionals will indicate that the settling party has fully satisfied its settlement agreement obligations.

Systems of record

The forensic professionals’ review is conducted by accessing and reviewing various data and documentation within the settling party’s SORs, which can also include its vendors’ SORs. In turn, access to the settling party’s records and supporting documentation, including confirmation of the settlement agreement population identification process through which the work papers are identified, is required and must be sufficient for the forensic professionals to substantiate and evidence the accuracy and validity of the work performed. This process includes capturing and saving within the work papers images from the settling party’s SORs, evidencing the relevant data and performing a review of any queries used to derive the population.

As the SORs are often heavily relied upon to conduct the necessary validation, forensic professionals are typically required to perform a quality review of the SORs before relying on them. Because the underlying information and processes are highly confidential and proprietary in nature, this presents a significant challenge for the parties to the settlement agreement.^[6] It is not uncommon for forensic professionals to rely on internal testing (e.g., internal reports and quality control testing) and external testing (e.g., reports on the organisation’s controls and regulatory reviews). Although often complicated and not without a fair amount of negotiation, it is a necessary step to confirm the validity and accuracy of the compliance process. This issue is often exacerbated when dealing with third-party vendors who are not direct parties to the settlement agreement. In that situation, the settling party relies upon the data, but typically does not have direct access to the third-party vendors’ SORs.

Identification of the full affected population

The independent monitor’s mandate may also include certifying the accuracy and completeness of the affected population subject to the terms of the settlement agreement. In so doing, the independent monitor typically engages forensic professionals to understand how the settling party determined the relevant population and to validate the accuracy and completeness of the identification of actions or items within the settlement agreement.

In some cases, the settling party agrees to provide the independent monitor’s forensic professionals with access to the relevant documentation and SORs that are necessary to enable the forensic professionals to verify the settlement agreement population. The SORs may include queries or coding, and other information reasonably expected to be relied upon, to enable the forensic professionals to understand the evidence that is needed to assess whether the settling party has accurately identified the population subject to the terms of the settlement agreement. In those instances, however, it may require that the forensic professionals obtain the supporting documentation (active and archived) from the settling party’s SORs. This approach includes the forensic professionals’ development of independent queries within platforms and the use of independent access to the source systems to extract the populations, as well as manual validation and a comparison of the information acquired with that identified within the settlement agreement.

Alternatively, the independent monitor’s forensic professionals may only be required to review the method by which the settling party determined the number of actions or items approximated in the settlement agreement. This means that the forensic professionals only assess the means and mechanisms used by the settling party as part of the process to identify the population subject to the settlement agreement. Specifically, the forensic professionals will evaluate whether the settling party’s population identification work papers are substantially accurate and independently assess whether the results of the population analysis effectively substantiate and confirm the accuracy and validity of the population identified within the settlement agreement population. In such a case, the settlement agreement would require the settling party to provide the independent monitor’s forensic professionals with documents and information relevant to the settlement agreement that are reasonably necessary to complete the independent monitor’s mandate.

In any event, a constant consideration and focus of all parties to the settlement agreement is needed to provide the necessary information to support testing under the work plan and metric testing template in a secure platform that protects the integrity and confidentiality of the information, yet still provides sufficient visibility and detail to accurately validate completeness and compliance with the settlement agreement requirements. The settling party’s records and related systems, which reflect its operations, are typically maintained within internal and external platforms. As such, the forensic professionals will typically hold meetings with representatives of the settling party and its vendors on multiple occasions to obtain an understanding of the functions, processes, applications and proposed approaches for testing and validation, including the retrieval process for documentation across enterprise systems and archives.

Metrics

Certain settlement agreements specifically identify metrics by which to assess compliance with the requirements of the settlement agreement. These metrics address distinct matters within the settlement agreement that may have unique complexity or specific time frames (or both) or be associated with a specific population that is the subject of the settlement agreement.^[7] Metrics are contemplated and designed to specifically evaluate whether the settling party is fulfilling its distinct, identifiable obligations under the settlement agreement for individual, unique matters. The metrics are mapped to the terms of the settlement agreement and are implemented via the standards established within the work plan and respective metric testing templates.

The population and testing methodologies for each metric are generally set forth in the work plan, which includes the documentation required for validation, and the development of detailed metric testing templates for the independent monitor’s forensic professionals to use in reviewing work papers in connection with confirmation of the same. These metric testing templates set forth the specific rules for how different actions or items are identified, the eligibility of these actions or items for compliance with the terms of the settlement agreement, and how remediation will be determined.^[8]

Identification of the metric populations

To select the relevant population for each metric, the settling party will need to evidence the mechanisms by which the identified metric population is extracted and identified from within the full settlement agreement population or from within the settling party’s systems, including its vendors’ SORs, if applicable. These queries are performed by the independent monitor’s forensic professionals, whose review and evaluation also include the preparation of documentation that evidences that the respective metric populations conformed in all substantial respects to the settlement agreement.

In those matters in which the metric populations are extracted and identified from within a consistent and verifiable settlement agreement population, it is not uncommon for the settling party to create a controlled environment for population identification. In such a case, metric level population testing and validation can be conducted without affecting the settling party’s ‘live’ SORs. In those cases, it is imperative for the forensic professionals to validate the integrity of the data and the accuracy of the full settlement agreement population prior to applying additional rules associated with metric population identification and validation. Typically, the settling party’s internal infrastructure and technology team will construct a controlled population environment for identifying the populations that are subject to manual item level testing by the independent monitor’s forensic professionals to ensure accuracy and completeness of each metric population.

The forensic professionals will then identify and extract metric populations for testing from the controlled population environment. This process includes the development of queries within the controlled population environment to independently extract metric populations, and to carry out further data analysis and comparisons to the entire settlement agreement population to ensure that all actions or items, as well as fields necessary for validation, were accurately captured. This analysis includes the forensic professionals investigating actions or items that were not included in the metric population but have been identified as meeting the requirements for inclusion.

The forensic professionals document their analysis and the logic used to identify the metric populations from the settlement agreement population.^[9] In addition, it is not uncommon to apply a threshold error rate to determine whether the settling party’s assessment with respect to each metric population is accurate. Using this approach, if the metric population (as identified by the forensic professionals) was greater than the estimated metric population (as provided by the settling party) by an agreed percentage (as identified within the work plan or the metric testing template), the settling party would ‘fail’ with regard to the identification of the applicable metric population, and the appropriate course of action (as identified in the work plan) would be required.

Sampling

In certain circumstances, the independent monitor’s forensic professionals’ approach to evaluate the settling party’s compliance with the metrics eligibility testing and determination of remediation is performed on a sample basis, rather than a review of every action or item in the population.^[10] In such cases, the forensic professionals use statistical parameters and a sampling methodology explicitly delineated within the work plan or respective metric testing template. Based on these statistical parameters and sampling methodology, the forensic professionals select a sample of actions or items from the applicable population. The samples selected by the forensic professionals from the appropriate populations will be based on a detailed review of the settling party’s relevant records and include a number actions or items that are statistically significant.^[11] As such, the forensic professionals prepare detailed work papers reflecting its population and sampling review and analysis, which includes verification of the sampling tool used by the forensic professionals and validation of other relevant sampling methodologies.

Sampling is a scientifically valid method of using a subset (sample) of a group (population) about which one wishes to draw inferences to estimate some property or characteristic of that population without having to examine each item of the population.^[12] It involves drawing a random sample from a population in which each element item within the population has an equal probability of being selected and included in the sample. The randomness of the sample allows one to extrapolate the sample results to the population because it allows a statistician to assume that the relationships in the sample are consistent with the relationships in the population.^[13]

Extrapolating from the sample to the population involves measures of reliability and precision, known as ‘confidence levels’ and ‘margins of error’. The confidence level is the probability that the population value will fall within a specified range around the value extrapolated from the sample.^[14] The margin of error is synonymous with sampling error and is typically most commonly referenced in political polls.^[15] As a simple example, assume that a population of 35,000 actions are identified within the settlement agreement as eligible and entitled to receive remediation. Rather than testing each of the 35,000 actions in the settlement agreement population, a completely random sampling methodology with a 95 per cent confidence level and a margin of error of plus or minus 5 per cent indicates that a randomised sample size of 380 actions would enable a tester or reviewer to obtain an accurate, reliable representation as to the settlement agreement population as a whole.^[16]

Sampling selections

The independent monitor’s forensic professionals will reach their conclusions by randomly selecting statistically valid samples from actions or items that the settling party deemed eligible and sought remediation pursuant to the settlement agreement. Each of the metric samples is drawn from a separate and distinct metric category population identified within the work plan and metric testing template, and each is treated as a testing population. These random samples are typically selected by using an established, licensed analysis software product (e.g., Microsoft SQL Server Management Studio or Microsoft Excel). In determining the sample size, the forensic professionals, in accordance with the work plan and metric testing template, use the ‘confidence level’ and ‘margin of error’ methodology delineated in the work plan and metric testing template, and as described in the previous section. Through this process, the forensic professionals identify a statistically significant number of actions or items for testing and review.

At that point, the metric population is randomised and each action or item is assigned a computer-generated random number. Using the sample size calculator, a certain number of random actions or items are selected for testing.^[17] These are then typically loaded into a database from which the forensic professionals’ testing is performed. The forensic professionals’ work papers will include screen shots of each step to evidence that the sampling methodologies were carried out properly.

Testing process

For each action or item, the independent monitor’s forensic professionals determine whether it met the criteria described within the settlement agreement and based on the assembled data for that action or item. For any action or item determined not to meet the criteria of the settlement agreement, the forensic professionals would conclude that the affected population was not successfully remediated or did not meet the compliance requirement. For each action or item determined to meet the criteria for remediation, the forensic professionals ensure that the settling party had correctly reported the remediation completed by verifying key data points. In many work plans and metric definition templates, a tolerance level for error is also established. This serves to define what constitutes a ‘failure’ on a single action or item based on a pre-determined threshold (typically a count, an amount or a percentage) beyond which the variance between the outcome determined by the independent monitor’s forensic professionals and the outcome reported by the settling party is reportable as a ‘failure’.

In addition to verifying whether the remediation met the applicable criteria for the actions or items within the settlement agreement population, for settlements that include consumer relief, the forensic professionals often determine the amount of remediation within the population tested (actual remediation amount) and compare that with the amount of remediation claimed by the settling party for the same actions or items (reported remediation amount).^[18] If the amounts are equal, or if the actual remediation amount is within a certain tolerated percentage^[19] of the reported remediation amount, as identified within the work plan or metric testing template, the reported remediation amount will be deemed correct and confirmed by the forensic professionals. If, however, the forensic professionals determine that the reported remediation amount for the population exceeds the actual remediation amount by more than the tolerated percentage established within the work plan or metric testing template, that population would ‘fail’ the forensic professionals’ review and this failure would be communicated to the independent monitor and then to the settling party.

When there is a ‘failure’, the settling party is required to implement corrective measures on the failing population, as determined in the work plan. Generally, however, when forensic professionals determine that the settling party has performed remediation on a larger population than was stipulated (or greater consumer relief than required) in the settlement, independent monitors may accept the remediation as reported.

Ultimately, the forensic professionals confirm the eligibility of the actions or items identified by the settling party and whether the remediation claimed is accurate and conforms to the requirements of the work plan and metric testing template.^[20] By so doing, the independent monitor can determine whether the settling party is in compliance with the settlement agreement.

Non-compliance

If the independent monitor’s forensic professionals conclude that the settling party has not yet met the settlement agreement’s standard or fulfilled the conditions required under the assessment, the independent monitor will be informed and will send the settling party a report of the assessment. Typically, the settling party can provide additional information to the independent monitor for consideration and can object to the assessment.

If, however, even with the additional information provided by the settling party, the independent monitor and his or her forensic professionals still believe the settling party has not yet met the settlement agreement’s standard or fulfilled the conditions required by the settlement agreement, the settling party will submit a corrective action plan to the independent monitor, for his or her approval. The independent monitor’s forensic professionals will validate or confirm the root cause of the error and determine whether the plan is sufficient to correct the error going forward. Once the corrective action plan has been vetted and approved by the independent monitor, the forensic professionals will determine whether the settling party has fully and successfully implemented the corrective action plan and, if applicable, remediated any harm caused by the failure. Typically, metric testing will halt during this period of vetting and approval. Only once remediated and tested will the settling party again submit actions or items for review and the forensic professionals can resume testing.

Conclusion

The independent monitor will ultimately issue a report that describes the independent monitor’s forensic professionals’ role in establishing, assessing and overseeing the settlement agreement. This report will explain the forensic professionals’ role in validating the applicable populations, confirming that the eligible actions or items were correctly identified and whether the appropriate remediation was provided. In summary, the forensic professionals are a vital part of an independent monitor’s team and mandate to ensure that a settling party adheres to its obligations within the settlement agreement.

Footnotes