Model Validation
Model Validation is a top concern of Compliance Officers as regulatory scrutiny has put Model Governance into the spotlight, requiring them to take accountability for model accuracy and performance. With the introduction of NY DFS 504 regulation and the ongoing discussion of the OCC Model Governance guidelines, banks have been under pressure to ensure that they understand their models. In addition, banks must be able to present sufficient documentation to demonstrate their model design, theory, accuracy, and logic; showing the intended use and applicability of each model
As such, banks are increasingly relying on assistance from consulting firms who possess a strong understanding of the financial compliance landscape. These consultants assist with evaluating the full end-to-end processes of their compliance models, including governance policies and procedures, conceptual soundness of design, assessment of data quality, model accuracy, and model performance.
Our Expertise
Matrix-IFS’ Model Validation services are provided by an advisory team which consists of some of the top minds in the financial crime space and ex-pats from the “Big4”. The team has a combined experience of over 40 years performing independent Model Validation engagements for various clients across many industry-leading compliance systems, including but not limited, to NICE Actimize, FIS, Fiserv, NetReveal, Mantas, SAS, Verafin, Banker’s Toolbox, Fircosoft, Accuity, and SafeWatch. To support our advisory teams, Matrix-IFS possess in-depth technical expertise stemming from 400+ transaction monitoring and sanctions implementation projects.
Why Perform Regular External Model Validations
- Model Validation helps financial institutions with timely identification of potential regulatory gaps. It prepares them to mitigate those risks with corrective actions by uncovering shortfalls/limitations in their existing compliance models.
- Often, financial institutions struggle to identify the measures necessary to validate the effectiveness of their methodologies, workflows, controls, model performance, alert disposition process, and adequacy/consistency of documentation throughout the compliance program.
- Having a Model Validation performed by an external entity provides an unbiased, critiqued insights into the institution’s compliance program.
Our Methodology
Matrix-IFS performs Model Validations based on a rigorous, well-defined, organized framework that is consistent with a comprehensive understanding of OCC 2011-12, SR 11-7, and DFS-504 guidelines. Below are the key pillars of the Model Validation framework:
- Governance, Policies, and Controls Review – To understand the organization’s controls in mitigating risks and identifying any potential gaps or opportunities for further improvement, Matrix-IFS reviews the financial institution’s policies, procedures, model governance, BSA/AML/OFAC risk assessment, roles/responsibilities, and internal controls documentation.
- Data Assessment – Identifying and assessing Critical Data Elements (CDEs) required by the monitoring rules/algorithms in place. Also, validating source and intermediary systems to target and asses the quality and completeness of the CDEs.
- Conceptual Soundness of Design – Based on the financial institution’s risk profile, perform a detailed review of applicable typologies and their respective mapping against the monitoring rules/algorithms in place, ensuring complete/intended monitoring coverage. When required, perform a customer-based segmentation analysis to determine the applicability of the monitoring models and proper risk targeting.
- Design Implementation Review – Perform independent testing of existing rules/algorithms by replicating or re-executing the solution in a separate environment following documented configuration settings and logic.
- Output Performance Assessment – Review the effectiveness of rules/algorithms in place by statistically assessing alerts. Based on the alerts assessment, identify any opportunities to improve efficiency by revising thresholds/configuration settings or revising the rule’s logic.
Types of Model Validation
Transaction Monitoring (TM)
Assess the applicability, design, effectiveness of active rules/scenarios and their corresponding thresholds used as part of the financial institution’s TM, system based on its risk profile (i.e., products and services, geographical exposure, transaction types, customers, etc.)
Sanctions
Assess the effectiveness of the Financial institution’s transaction/message filtering system by validating relevant test cases against live/current regulatory lists (both exact and fuzzy matching capabilities), internal blacklists or whitelists processes, and list management.
Customer Risk Rating
Assess whether the score factors used as part of the model’s risk scoring aligns with the financial institution’s risk profile and identify whether the risk weights are leading to the proper classification of customer risk. Review of the processes used to feed customer data back into the model for periodic scoring.
Tools & Accelerators
Having provided services for dozens of financial institutions, the Model Validation team has built a comprehensive set of in-house tools and accelerators such as:
- Data quality validation
- Entity resolution visualization
- Libraries of U.S and other jurisdiction Red Flags
- Tools to parse standard payment messages (SWIFT, ACH, etc.)
- Tools to parse standard format data files (XML, JSON, etc.)
- Exhaustive Data Quality assessment scripts library
- Library of scripts for rules/scenarios for industry-leading solutions
- Sanctions test case generator
- Sampling tools
Model Validation & Tuning Accelerators
Sample Customers
