The National Fair Housing Alliance
Purpose, Process, and Monitoring
Purpose
The National Fair Housing Alliance (NFHA) has proposed a new auditing framework called Purpose, Process, and Monitoring (PPM) to help assess the fairness and efficacy of algorithmic systems in housing and lending. The PPM framework is designed to provide an equity-centered roadmap for auditors to assess the potential risks that algorithmic systems may pose to consumers, institutions, and society, and to ensure that they are fair, equitable, explainable, and transparent.
Key Tenets
The PPM framework is composed of three stages: Purpose, Process, and Monitoring.
- The Purpose stage examines project goals as well as the expectations, requirements, and objectives of stakeholders who are trying to solve a business problem. During this phase, auditors are guided to seek information to make educated decisions about risks the business problem may pose to consumers, institutions, and society at large.
- The Process stage evaluates the design, theory, and logic used to develop an algorithmic solution in the context of business-use cases and design objectives. This stage includes five elements:
- Staff Profile: To ensure the model is built by a diverse and inclusive team that’s trained to spot and prevent issues that lead to unfavorable outcomes.
- Data Assessment: To audit data sources and data fields to determine if the data used is appropriate, representative, fair, and accurate.
- Model Assessment: To evaluate information about training algorithms, parameters, hyper-parameters, and any fairness constraints used during the development of the model.
- Outcome Assessment: To determine if the performance of the final model is in line with its original design objectives, which include minimizing risks to consumers, institutions, and society.
- Model Use and Limitation: To document known limitations and assumptions of the model, as well as identify where the model may or may not be used outside of its intended scope.
- The Monitoring stage focuses on the post-development phase of an algorithmic model’s lifecycle. This stage includes two main considerations:
- Ongoing Validation of Production Model: To ensure that the data used to train the model isn’t statistically different from the data encountered in the real world.
- Clear Defense and Security Measures: To protect the model against attacks that jeopardize consumer data or the system’s integrity, and ensure the defenses used guarantee fairness and accountability.