Why Candidates Love Feedback From AI

In today’s competitive job market, understanding a candidate beyond their resume is crucial for discovering untapped talent. Equally important is candidate experience where the screening process needs to be both inclusive and provide candidates with feedback.

Sapia.ai’s Smart Interviewer™ helps organizations achieve both; assessing a broader, diverse pool of candidates and providing them with a learning experience instead of a one-sided approach like uploading a resume and completing a form. Our Smart Interviewer™ achieves this with a text chat based structured interview that is untimed and asynchronous, and from which every candidate receives a ‘My Insights’ profile, with personality insights and coaching tips derived from their written interview answers.

As an AI-first platform, learning from feedback is at the heart of our product development. We request our candidates to provide optional feedback on the My Insights report they receive. With over 3 million candidates having already met our Smart Interviewer™, we have thousands of feedback comments to learn from. In the past, we have used traditional machine learning approaches such as clustering and topic modeling to find key themes in the feedback. However with the advent of generative AI, especially Large Language Models (LLM), tasks such as understanding themes in feedback comments, be it candidate or customer, have been made easier, and more effective.

This paper shares the themes we found when we asked Meta’s Llama-2 model to discover them in the candidate feedback on My Insights. We also briefly explain the approach we used, including the prompts. We think HR teams can use these generative AI tools like Meta’s Llama-2, Open AI ChatGPT and Anthropic Claude to gain further insights from data they already have.

Candidates Love My Insights

This is what we found through analyzing thousands of candidate feedback comments without bias. How did we do it? We passed the comments to an instance of the Llama-2 model and asked it to find themes in the candidate feedback (see section “How to use Llama-2 to find themes” below for the method and prompts).

Here are the final topics and summary of the findings from Llama-2 (verbatim).

Based on the feedback provided by job candidates who used My Insights, the following 10 themes emerged:

1. Self-awareness and personal growth: Candidates appreciated the opportunity to gain insight into their personality, strengths, and weaknesses, and found the feedback to be helpful in identifying areas for improvement and personal growth.

2. Accuracy and validity: Several candidates were impressed by the accuracy and validity of the insights provided by My Insights, finding that the feedback was consistent with their own self-perception and observations.

3. Confidence and self-esteem: The feedback comments often mentioned the candidates’ increased confidence and self-esteem after receiving the insights, which could potentially lead to better job performance.

4. Improving work relationships: Candidates appreciated the insights that helped them understand their colleagues and supervisors better, and found the feedback to be valuable in improving work relationships and communication.

5. Career development: Many candidates saw the insights as a valuable tool for career development, believing that the feedback could help them identify areas for improvement and develop strategies for career advancement.

6. Personalized coaching: Some candidates appreciated the personalized coaching tips provided by My Insights, finding them to be helpful and actionable.

7. Independence and objectivity: Candidates appreciated the independence and objectivity of the insights provided by My Insights, finding that the feedback was not influenced by personal biases or agendas.

8. Time-saving and efficiency: Several candidates mentioned that the insights provided by My Insights saved them time and effort in their job search, as the feedback was concise and to the point.

9. Positive attitude and outlook: Many candidates expressed a positive attitude towards the insights provided by My Insights, finding that the feedback reinforced their positive qualities and helped them maintain a positive outlook on their personal and professional lives.

10. Overall satisfaction: Finally, many candidates expressed overall satisfaction with the insights provided by My Insights, finding the feedback to be helpful, informative, and useful in their job search and personal development.

These themes suggest that My Insights is a valuable tool for personal growth, self-awareness, and professional development. The insights provided help candidates understand their strengths, weaknesses, and areas for improvement, leading to increased confidence, adaptability, and collaboration in the workplace. The themes also highlight the importance of communication skills, teamwork, emotional intelligence, and practical experience in the workplace. Overall, the feedback suggests that My Insights is a useful tool for job candidates to gain insight into their personality and behavior, and to identify areas for improvement and growth.

How to use Llama-2 to find themes

We utilized a straightforward yet effective methodology to sift through a large volume of feedback comments. Our goal was to perform a quick-fire analysis to unearth the major themes that resonate with the candidates regarding the My Insights feature.

Why did we use Llama-2 and not ChatGPT?

As a responsible AI company, data sharing with 3rd party applications is something we take seriously. While we are sure ChatGPT is equally capable of achieving the same, we use an internally hosted Llama-2 model for all our LLM related data analysis, which prohibits data sharing. You can use any suitable LLM fine-tuned on instruction following to achieve the same.

Approach in a nutshell

Given we have a large number of feedback comments that exceed the context window of Llama-2, we used an iterative approach that was automated with Python. The key steps of the approach are:

1. Sampling and chunking:
– Select a random sample of N=10,000 feedback comments. You can select any N depending on the size of the data set you have.
– Break down the data into manageable chunks, each containing 100 comments that fits the context window

2. Initial theme identification:
– Prompt the Meta Llama-2 model with a specific request to identify unique themes from the comments concerning the My Insights report.
– Following is the prompt we used. You can further improve it to obtain specific outputs.
– The number of 12 was picked arbitrarily here as a number that is greater than the 10 final topics we want to discover.

Top level prompt: Find 12 unique themes after carefully analyzing the feedback comments below. These feedback comments are from job candidates on the personality insights called My Insights they received from Sapia.ai chat-based interview tool.Provide a short description of each theme you find.

List of feedback comments:

3. Iterative theme summarization:
– Post initial identification, we honed down the themes further to a crisp list of 10 overarching themes.

Summarization prompt: Strictly find 10 unique themes after further analyzing the feedback themes below. These feedback themes are from job candidates on the personality insights called My Insights they received from Sapia.ai chat-based interview tool.

List of feedback themes: While we are sure these prompts can be further fine tuned (as we have done in some of our subsequent analysis), you can see how two simple prompts can be used to gain insights into a large volume of feedback data. Once you find the themes, you can also ask the Llama-2 to assign the themes to all the feedback comments in a separate task that can be used for quantitative analysis such as topic prevalence.

Final thoughts

You can see how a quick-fire analysis of large volumes of feedback data can be performed with an LLM like Llama-2 to gain valuable insights into candidate experiences. Instead of looking at word clouds from a clustering tool, now you can find summaries of the themes as if a human analyst has read and grouped them. Further, you can keep asking for various insights, with natural language prompts.

In our case, the extracted themes from candidate feedback on My Insights show an overwhelmingly favorable experience, highlighting how My Insights help candidates understand and further improve soft skills and work relationships.

It is important that HR professionals employ these generative AI tools in responsible ways to learn from feedback to continually refine the recruitment process, making it a win-win for both employers and candidates.