Sapia.ai (formerly Predictive Hire) is a leading provider of artificial intelligence powered employment application screening tools, including Chat Interview, a text-based interview tool in which applicants provide textual responses to standard interview questions and then are scored based on these responses.

Sapia.ai engaged BLDS, LLC, a nationally recognized statistics and economics firm, to perform a Disparate Impact Analysis (DIA) of Sapia.ai’s Chat Interview tool for purposes of conducting a bias audit.

This report covers BLDS’s DIA of Sapia’s Chat Interview models for North America. The models tested were developed and deployed in two jurisdictions: the United States and Canada.

A Chat Interview model provides a score ranging on an interval from 0 (representing a poor fit) to 1 (representing a strong fit) for applicants for a specific employment opening. The Chat Interview model further provides a ‘Yes’ recommendation for the top 40% of the applicants, a ‘Maybe’ recommendation for the middle 20% of applicants, and a ‘No’ recommendation for the bottom 40% of applicants. The score and recommendations are intended to provide a tool for employers to screen applicants and to prioritize which applicants to move forward in the interview process. Employers using Chat Interview are free to use the recommendations and/or the scores however they see fit. Because employers may use the Chat Interview scores and recommendations in various ways, BLDS employed a wide range of DIA tools and metrics to uncover any disparate impact that could arise across numerous different use cases.

At an overall level, BLDS analyzed model scores using two commonly used metrics for determining whether there is evidence of practically significant disparate impact: the Standardized Mean Difference, or SMD, and the Adverse Impact Ratio, or AIR.
Using the SMD, BLDS performed a total of 23 tests for protected groups on the North American models. Under the SMD test, BLDS found no evidence of practically significant disparate impact for any protected group assessed in the United States or Canada.

Using the AIR, BLDS performed a total of 49 tests for protected groups on the North American models. First, the use case where applicants advanced as the result of a ‘Yes’ recommendation was tested using the AIR. Second, the use case where applicants who received a ‘Yes’ or ‘Maybe’ recommendation was tested using the AIR. None of these tests revealed evidence of practically significant disparate impact for any protected group in the United States or Canada.

For more information, contact Barb Hyman here.