AI Generated Content in Online Text-based Structured Interviews

Study Details

• Data Sample: Over 573,500 candidate responses on Sapia.ai Chat Interview™ platform in 2023.
• Analysis Process: A detector based on GPT-2 was developed to identify AI-generated content by comparing the formulaic nature of AI text to the randomness of human-written text.

Key Findings

• Detection Accuracy: The AGC classification model is highly accurate, achieving an overall accuracy score of 94.49%.
• Prevalence Trends: Only 1.16% of candidates were flagged for using AGC across all five questions, while 20.20% had at least one response flagged.

Deterrence Methods: Introducing upfront messaging, disabling pasting, and adding in-chat warnings reduced the prevalence of AGC usage.

Key Takeaway
In the age of ChatGPT, implementing an AI Generated Content (AGC) detector and effective deterrence strategies can significantly help maintain the integrity and fairness of text-based interviews and assessments. This creates a more equitable and accurate evaluation process, ensuring that candidates responses reflect their true knowledge, skills, and abilities. For more information, read the full paper here.