Measuring disparate impact in human and machine decisions

Abstract

Empirical analyses have grown increasingly important in discrimination litigation with the greater availability of detailed data on individuals and decisions. A popular analytic strategy is to estimate disparities after adjusting for observed covariates, typically with a regression model, in hopes of ferreting out discriminatory intent. This approach, however, is ill-suited to auditing algorithms that are now commonly used to aid decisions, which typically do not include race or other legally protected factors as inputs. Motivated by legal understandings of disparate impact, we introduce an approach that aims to measure “unjustified” disparities in both human and machine decisions. Our method, which we call risk-adjusted regression, proceeds in three steps. In the first step, we combine all available information in a machine learning model to estimate the value, or inversely, the risk, of taking a certain action, such as approving a loan application or hiring a job candidate. Second, we measure disparities in decisions after adjusting for these risk estimates alone. Finally, in the third step, we assess the sensitivity of results to potential mismeasurement of risk. We demonstrate this approach on a detailed dataset of 2.2 million police stops of pedestrians in New York City, and show that traditional statistical tests of discrimination can substantially understate the magnitude of (risk-adjusted) racial disparities.