Recruitment Software

An AI hiring firm says it can predict job-hopping based on your interviews. The idea of “bias-free” hiring, already highly misleading, is being used by companies to shirk greater scrutiny for their tools’ labor issues beyond discrimination.

Since the onset of the pandemic, a growing number of companies have turned to AI to assist with their hiring.

The most common systems involve using face-scanning algorithms, games or other evaluations to help determine which candidates to interview.

Activists and scholars warn that these screening tools can perpetuate discrimination. However, the makers themselves argue that algorithmic hiring helps correct for human biases.

Algorithms can be tested and tweaked, whereas human biases are much harder to correct—or so the thinking goes.

In a December 2019 paper, researchers at Cornell reviewed the landscape of algorithmic screening companies to analyze their claims and practices. Of the 18 they identified with English-language websites, the majority marketed as a fairer alternative to human-based hiring. Thus suggesting that they were latching onto the heightened concern around these issues to tout their tools’ benefits and get more customers.

But discrimination isn’t the only concern with algorithmic hiring. Some scholars worry that marketing language that focuses on bias lets companies off the hook on other issues, such as workers’ rights. A new preprint from one of these firms serves as an important reminder. “We should not let the attention that people have begun to pay to bias/discrimination crowd other issues,” says Solon Barocas, an assistant professor at Cornell University and principal researcher at Microsoft Research, who studies algorithmic fairness and accountability.

The firm in question is Australia-based Sapia (Formerly PredictiveHire), founded in October 2013.

It offers a chatbot that asks candidates a series of open-ended questions. It then analyses their responses to assess job-related personality traits like “drive,” “initiative,” and “resilience.”

According to the firm’s CEO, Barbara Hyman, its clients are employers that must manage large numbers of applications, such as those in retail, sales, call centers, and health care.

As the Cornell study found, it also actively uses promises of fairer hiring in its marketing language. On its home page, it boldly advertises: “Meet Smart Interviewer – Your co-pilot in hiring. Making interviews super fast, inclusive and bias free.

As we’ve written before, the idea of “bias-free” algorithms is highly misleading. But Sapia’s latest research is troubling for a different reason. It is focused on building a new machine-learning model that seeks to predict a candidate’s likelihood of job-hopping. That is the practice of changing jobs more frequently than an employer desires. The work follows the company’s recent peer-reviewed research that looked at how open-ended interview questions correlate with personality.

The study used the free-text responses from 45,899 candidates who had used Sapia’s chatbot.

Applicants had originally been asked five to seven open-ended questions and self-rating questions about their past experience and situational judgment.

These included questions meant to tease out traits that studies have previously shown to correlate strongly with job-hopping tendencies, such as being more open to experience, less practical, and less down to earth. The company researchers claim the model was able to predict job hopping with statistical significance. Sapia’s website is already advertising this work as a “flight risk” assessment that is “coming soon.” Sapia’s new work is a prime example of what Nathan Newman argues is one of the biggest adverse impacts of big data on labor.

Machine learning for the win!

Machine-learning-based personality tests, for example, are increasingly being used in hiring to screen. This is to out potential employees who have a higher likelihood of agitating for increased wages or supporting unionisation. Employers are increasingly monitoring employees’ emails, chats, and data to assess which might leave and calculate the minimum pay increase to make them stay.

None of these examples should be surprising, Newman argued. They are simply a modern manifestation of what employers have historically done to suppress wages by targeting and breaking up union activities. The use of personality assessments in hiring, which dates back to the 1930s in the US, in fact began as a mechanism to weed out people most likely to become labor organizers. The tests became particularly popular in the 1960s and ’70s once organizational psychologists had refined them to assess workers for their union sympathies.

In this context, Sapia’s fight-risk assessment is just another example of this trend. “Job hopping, or the threat of job hopping,” points out Barocas, “is one of the main ways that workers are able to increase their income.” The company even built its assessment on personality screenings designed by organizational psychologists.

Barocas doesn’t necessarily advocate tossing out the tools altogether. He believes the goal of making hiring work better for everyone is a noble one and could be achieved if regulators mandate greater transparency.

By Karen Haoa, July 24, 2020, MIT Technology Review | https://www.technologyreview.com/

Join the movement

To keep up to date on all things “Hiring with Ai” subscribe to our blog!

Finally, you can try out Sapia’s Chat Interview right now, or leave us your details here to get a personalised demo.

Big decisions on ‘gut feel’

It scares me sometimes when I think about the big decisions I’ve made on gut feel and will probably continue to make relying on my instincts.

Personally, I would love to be armed with meaningful data and insights whenever I make important life decisions. Such as what’s the maximum price I should pay for that house on the weekend, who to partner with, who to work for, and who to hire into my team. Data that helped me see a bigger picture or another perspective would be very valuable. For most of those decisions there is so much information asymmetry which makes it feel even riskier. For sure I could check out glassdoor when choosing my next job but it comes with huge sample bias and not much science behind it.

So why is there still an (almost) universal blind acceptance that these decisions are best entrusted to gut feel? Especially given the facts show we are pretty crap at making good ‘gut’ based decisions.

Dialling down subjectivity

I’m one of those people that believe in the power of AI — to remove that asymmetry, to dial down the bias, to empower me with data to make smarter!

At a recent HR conference, a quick pulse around the room confirmed there is high curiosity and appetite to understand AI. What we’re missing is the clarity about the opportunities and what success looks like from using it. The concern about how to navigate the change management exercise that comes with introducing data and technology into a previously entirely human-driven process is daunting.

The best human resources AI is not about taking the human out of hiring and culture decisions. Far from it. It’s about providing meaningful data to help us make better decisions faster.

Having worked in the ‘People and Culture’ space for a while, I know building trust in how the organisation makes decisions — especially people decisions — is hard in the absence of data. Yet we all know that transparency builds trust. So how can you build that trust through transparency when the decision-maker is a human — and the humans make decisions in closed rooms and private discussions.

Human decision making is the ultimate black box.

Remember that feeling when the recruiters call up and say you weren’t a good fit — who feels great about that call? A total black box cop-out response!

It doesn’t have to be this way, and the faster we can get to better decision making the better. Seven months ago, I joined a team of data scientists who had spent the prior three years building technology that relies on AI to work its magic and equip recruiters with meaningful and actionable insights when hiring.

I’m no data scientist. I have had to learn the ins and outs of our AI pretty fast. And because our technology is at work in the people space, I’m learning how to ensure the AI is safe, fair and our customers trust it and us to do the right thing with it.

If we reduce it to its core process, a machine learning algorithm is trying to improve the performance of an outcome based on the input data it receives. In some instances, such as in deep learning algorithms, it’s trying to simulate the functioning of the human brain’s neural networks, to figure out the patterns between the data inputs and data outputs.

A machine is infinitely more obliging and easier to understand than a human in responding to the direction.

Because it has no feelings, it’s going to be free of the biases humans bring to these critical decisions. Plus machines are more malleable to learning and way faster at it. This is more critical these days when roles are changing dynamically and swiftly as industries are disrupted.

Our team plays in predictive analytics for recruitment space. What this means is our AI seeks out the lead indicators of job success: the correlating factors between values, personality and job performance. We all intuitively know that behaviours drive leading indicators. But we struggle to assess for those consistently well.

Our job is to augment your intelligence and ability to make the right decision. By knowing how people treat others, what drives them, and their values, you become better informed about the real DNA of a person and how they might function in your team.

All of our customers are looking for a slightly different kind of worker and use AI to help.

A powerful motivator to use AI is to build confidence and trust in the process from both candidates and people leaders by dialling down the human element (getting rid of the bias) and revealing the patterns for success. Less room for bias = more fairness for candidates = more diverse hiring. Key to this is we don’t look at any personal information — the machine doesn’t know or care about your age, gender, colour or educational background.

For our customers having this data is empowering and helps them make smart decisions. For all the people who are affected by those decisions, they can feel relieved that they were considered on their merits, not based on someone’s gut feel.

There is no perfect method for sure to make the right decision.

But if I have to choose between trusting biased humans and (a sometimes) biased machine they create, I know which one I would trust more. At least with a machine, you can actually test for the bias, remove it, and re-train it.

Suggested reading:

https://sapia.ai/blog/hr-job-manage-risk/

With so many candidates in the market, it’s more important than ever to create an engaging and human candidate experience. But you need to balance that with finding the best talent for your role.

Skill testing can give recruiters a competitive advantage in today’s job market. Candidates who are hired on merit, rather than background, tend to stay longer and perform better over the long term. Here’s how to use skills assessments to fill your open positions, no matter how many applicants you are dealing with.

Your Guide to Skill Testing

• What is a skill test?
• How skill testing works
• How to run a skill test
• Using skill tests during hiring
• Skill test examples and templates
• Do new hires like doing skill tests?
• What are the benefits of a skill test?
• Do skill tests work?
• Are skill tests valid?
• Skill tests vs. interviewing
• Additional resources

What is a Skill Test?

A skills test is an assessment used to provide an unbiased, validated evaluation of a candidate’s ability to perform the duties listed in the job description.

Typically, a skills test asks a variety of questions in different formats to see how candidates perform on-the-job tasks. A good skills test includes questions that are capable of being answered by someone already doing the job and can accurately measure key performance metrics. Questions should also be specifically tailored to relate to the responsibilities of an open position. Many skills tests include immersive experiences, like coding challenges or job simulations, to mimic how a candidate performs when faced with a real-life scenario.

Other types of job-readiness evaluations deploy validated psychometric assessments to identify those in-demand soft skills: things like motivation, conscientiousness, resilience, and emotional intelligence. A personality assessment varies from a skills test in that it predicts how a person will behave in a specific scenario, rather than their ability to complete a task.

Related: Should You Use Psychometric Tests for Hiring?

While skills test cover task-related abilities, like coding, copywriting, or sales, some pre-employment assessments integrate the less tangible capabilities – things like teamwork and leadership. These qualities are sought after by executives at more than 900 companies, according to a Wall Street Journal survey of executives.

Yet, 89% of those surveyed said they have a “very or somewhat difficult time finding people with the requisite attributes.” Where traditional hiring methods fall short, a skills test can easily clarify a candidate’s true talent.

“Many service companies, including retailers, call centers, and security firms, can reduce costs and make better hires by using short, web-based tests as the first screening step. Such tests efficiently weed out the least-suitable applicants, leaving a smaller, better-qualified pool to undergo the more costly personalized aspects of the process.”

Research by John Bateson, Jochen Wirtz, Eugene Burke and Carly Vaughan via Harvard Business Review

Overall, skills tests can play a critical role in predicting on-the-job success. More so than resumes or job interviews, a skills test can assess the true potential of a new hire to go the distance with the company. Here’s how skill testing works, and why more companies than ever are starting to integrate skill testing into the recruitment and hiring process.

How Skill Testing Works

Skill testing works best when the questions being asked are specifically crafted to the role and needs of the team hiring the new candidate. In designing a skills test, combine different types of questions to get a 360-degree view of how a candidate will perform in different scenarios.

There are a variety of ways to set up a skills test – and we’ll get into the mechanics of how to actually run the assessment in the next section. But, designing a thoughtful aptitude test takes some initial foresight on behalf of the hiring manager and team.

Research by Deloitte suggests this sample process for selecting and implementing skill testing questions:

1. Define the “human elements” needed to perform the job
2. Compile questions that will measure and predict these human elements
3. Use the data gathered by the skills assessments to empower the next round of the screening process
4. Post-hiring, evaluate the efficacy of the hiring assessment to ensure the questions delivered the best result.

Ultimately, the best use for a skills assessment is to help recruiters move away from the resume and allow candidates to prove they are the real deal. Crafting the right series of questions should be a collaborative process between the recruiting team and the team hiring the new employee. Here’s how these teams can set up and run a skills test.

How to Set Up and Run a Skill Test

In designing a skills test or pre-employment assessment, there are a few specific steps to take in order to thoughtfully structure your questions.

Related: 5 Steps to Creating an Engaging Skill Assessment

Based on our work with over 8,000 customers, we recommend following these best practices in setting up and running your skills test. These tips can help with candidate engagement and lead to high rates of completion.

• Your skills test should include a minimum of six questions; somewhere in the eight to ten range is best.
• At least a few questions should require text answers; start with a text-based response in the first question, rather than a video or immersive question.
• Include an “immersive” style question, in which the candidate edits a document, spreadsheet, or presentation.
• To retain a candidate over the entire experience, start with easier questions and build up to more difficult ones later in the assessment.
• Try to minimize the use of timers to account for technical difficulties and give the candidate the best chance of success.

We also suggest that video responses not be timed; there are too many technical issues that can result from a candidate trying to film a one-way video interview. If you do wish to set a time limit, make sure it’s at a minimum of five minutes.

Running a skills test through Vervoe, or any other platform, is relatively straightforward. Vervoe’s skills assessments let you select questions from a library of assessment tools, or design your own questions based on the specific needs of your company. The Expert Assessment Library offers questions and trials created by experts in their fields, meaning they have at least 3+ years of experience in their specific area of expertise. You can preview questions from any of the assessments and add them seamlessly through the Vervoe platform.

Now that you know how to set up an assessment, when should you deploy this tool during the hiring process?

Using Skill Tests During Hiring

Timing is everything when it comes to adding a skill assessment to your hiring process.

Research by Harvard Business Review revealed that skills tests should come early in the hiring process. According to their study, “Many service companies, including retailers, call centers, and security firms, can reduce costs and make better hires by using short, web-based tests as the first screening step. Such tests efficiently weed out the least-suitable applicants, leaving a smaller, better-qualified pool to undergo the more costly personalized aspects of the process.”

Skill tests should be used to screen candidates in, not out. The issue many recruiters face is that the volume of candidates makes it impossible to carefully consider each person’s ability. Smart algorithms and AI tools can turbo-charge candidate assessments by scoring results quickly and removing human bias from the equation.

Vervoe’s algorithm scores candidates using a multi-layered approach. Candidates are ranked based on how well they performed, rather than filtered out if they didn’t achieve a certain benchmark. The top candidates easily rise to the top; but no one misses out on being considered for the next round. When used early in the hiring process, skill tests can select a more diverse pool of applicants to continue onto the next phase.

Skill Test Examples and Templates

There are many ways to set up a skills test, depending on the position for which you are hiring. Pre-employment skills tests can cover a range of positions: administrative assistant, finance and accounting, and call center reps are just a few roles that companies hire for using skills assessments.

Excel skill tests, coding skill tests, typing skill tests, and other computer skill tests are the most common forms of pre-employment assessments. Some companies focus on questions that are task-related, e.g. “Create a Powerpoint Slide that has a video embedded in the presentation.” Questions can get hyper-specific to test a niche skill, like a coding language, or be posed more broadly to test the general requirements for success at a certain level.

Some companies choose to focus on verifying the skills that will help a candidate succeed beyond the immediate position. This approach skews closer to a pre-employment assessment, with questions designed to reveal if a candidate can climb the corporate ladder, adapt in a challenging work environment, or respond under pressure.

For example, one call center rep test included questions such as, “You have an elderly customer on the phone who is having trouble understanding your instructions. A colleague is also trying to transfer a call from a customer you served before, and you have a scheduled follow-up call happening in 5 minutes. How would you handle and prioritize in this situation?”

Multiple choice, open-ended questions, and pre-recorded video responses are all great ways to see if a candidate has what it takes to do the job well. But, do candidates enjoy answering these types of questions?

Do new hires like doing skill tests?

By most accounts, candidates appreciate the opportunity to showcase what makes them great at their job. Orica, the world’s largest provider of commercial explosives, integrated skill-testing into their interview process to the delight of their job candidates. In revamping the interview process for graduate students looking to join the Orica team, recruiters consolidated their online evaluation components into one platform, Vervoe. The skill assessment combined questions focusing on skills, logic, and values.

An average of 86% of candidates completed the online process, and the reviews were mostly positive. Here’s what the candidates had to say about the skills test:

“The tests required total engagement and thought, and were a clear demonstration of what makes Orica different from any other company.”

“I think the questions were very diverse and it allowed me to showcase myself, my skills and abilities in different ways.”

“It gave me an opportunity to showcase who I am as well as challenge my skills”

This is just one example of how a skill test can change the entire interview process for a potential new hire. In a job market where people spend an average of 11 hours a week looking for a new job, it’s easy to get burned out, fast. Every job description starts to look the same; every interview begins to feel stale.

When given the opportunity to showcase their talent through real-world tasks, job candidates will jump at the chance to be engaged with the job description, rise above their resume, and challenge themselves. Companies that use Vervoe’s assessments experience a 97% candidate completion rate, which is among the highest engagement rates in the industry. Candidates love the opportunity to stand out from the crowd. Even if they aren’t hired, skills testing offers a break from the repetition of the stale interview experience.

What are the benefits of a skill test?

The benefits of a skills test aren’t limited to the candidate experience.

Recruiters looking to hire diverse, high-performing teams with better efficiency and consistency can use pre-employment tests to their advantage. Skills tests are a better predictor of performance than resume screenings or traditional interviews alone. Resume screenings are bad for three reasons. First, studies suggest that it’s common for candidates to lie on their CV. The person you think you’re hiring may not actually possess the qualifications you think they do.

“We just wouldn’t be able to interview 2000 people in two weeks. But what we could do is utilize Vervoe to more accurately and in quite an unbiased way, assess everybody’s application during that period.

Rather than just assess the first 200 [applicants] and maybe hire 150 of them, Vervoe allowed us to actually assess all 3000 applicants in a two week period and still be able to select the best 150.”

Jeremy Crawford, Head of Talent Acquisition at Medibank

Second, resumes only provide a high-level view of a candidate’s credentials and work experience. These items don’t offer qualitative insight into actual on-the-job performance. Coupled with recruiting biases that are built into the process, the third threat is that recruiters are privileging candidates based on background and demographics, rather than talent. Perhaps this is why new hires crash out as often as they do. According to one study, 46% of new hires “fail” within the first 18 months of being hired.

Skill tests can help take some of the bias out of the interview process, give recruiters a new evaluation metric to consider, and lead to happier, long-term hires. There’s ample evidence to suggest they really do work better than many of the other traditional hiring methods recruiters have relied on in the past.

Related: How to Avoid the 12 Kinds of Hiring Bias

Do skill tests work?

In our experience, skill testing works better than traditional hiring methods – with some caveats.

Without a doubt, aptitude tests can be used to replace resume screening. This style of sorting through candidates increases the chance that the best candidates will be unfairly eliminated. Good people get screened out, rather than screened in. So-called “pedigree proxies” – resumes and cover letters – are not indicative of job performance, yet they are often the quickest way a recruiter or algorithm can think of to cut down on their stack of candidate resumes.

Skills tests improve time to hire while allowing the hiring manager to see how someone will do the job, before they get the offer. This reduces turnover costs, which add up quickly: the cost of making the wrong hire can be up to 2.5x salary, easily over $100,000. Working with Vervoe’s skills assessments, on the other hand, can help a recruiter identify the best people at under $100 per hire.

The best skills tests, however, need the right formula to help the candidates succeed. Some recruiters focus narrowly on the skills that will help a new hire succeed in the immediate position for which they are hiring. Yet, many CEOs emphasize the importance of soft skills – things like leadership and teamwork.

Related: 5 Ways To Turn Rejected Candidates Into Allies

New hires may end up being disappointed and leaving because they lacked the soft skills needed to adapt to their new team, not necessarily the skills to perform the job. Recruiters must integrate questions into their skill assessment that focus on critical soft skills that predict long-term success. These validated psychometric assessments are key to assessing “culture fit” without defaulting to recruiter bias.

Is skill testing valid?

With any kind of assessment, there’s a common concern that’s quite commonly raised: is this assessment valid?

In summary:

• Any test that directly mimics what a person will do on the job can be considered “validated.”
• Tests of personality and soft skills are a riskier prospect even when they are “validated,” because they often lack the proper validation required to be EEOC compliant.
• Positive candidate experience and perceived fairness are two of the primary reasons why skill testing is an effective and expedient hiring practice.

There are many types of validity, and it’s rare that a test will satisfy every type. Looking specifically at tests for finding job fit, there are a few different types of validity that are particularly relevant, not just to ensure that the hire is a good one, but to ensure compliance with EEOC regulations.

Types of Validity

• Face validity
  The most basic form of validity, and sometimes the only one that can be obtained when a test is first created. Face validity essentially asks whether the test looks like it’s assessing what it claims to measure.One example is testing someone’s arithmetic skills. A set of math problems would have more face validity in this instance than, say, a word problem because a word problem is assessing both arithmetic skills and comprehension.
• Content validity
  For skill tests used in recruitment, the question of validity should be most focussed on this kind of validity. Content validity asks whether the test covers the full range of the construct that it’s supposed to measure.This means that in any assessment, the group of questions being asked needs to cover a wide enough range of skills, so that the person evaluating can be sure that the results show the candidate is capable of doing the tasks required on the job.
• Construct Validity
  When people ask if a test is “validated” or has “psychometric validity,” this is the kind of validity that they’re usually talking about. Construct validity asks whether the test actually measures the theory-based construct that it claims to measure.So, if you’re testing for general cognitive ability or personality, construct validity is absolutely essential, because they are indirectly related to whether someone can perform the job.But when it comes to testing skills that used directly on the job, face and construct validity are far more important.
• Predictive/External Validity
  This kind of validity is about whether or not the assessment predicts performance in other situations. So, if someone scores highly on the test, does that mean they’ll perform well on the job?There’s a big difference between tasks that are assessed without context, and tests reflect the day-to-day skills and tasks someone would need to have to perform the role.

Related: Skill Testing Validity

In all cases where assessments are used, and in every step of the recruitment process, it’s essential that employers track and remain aware of differences in performance that are biased toward particular demographic factors. At Vervoe, we constantly monitor assessments to make sure candidates take tests that are fair, and based solely on skills that reflect how they would perform on the job.

Skill tests vs. interviewing

In conclusion, we’ll leave you with few thoughts on skill tests compared to interviews.

First, interviews, in general, need a total overhaul. Recruiters have been asking the same, outdated interview questions for decades. Many candidates get overwhelmed by the performance anxiety inherent in the interview and may make (forgivable) mistakes. Nevertheless, many recruiters like the security of meeting someone before making an offer.

Many recruiters seek the same insight from a group interview or case study that they would get from an individual skill test. Unfortunately, using these methods can’t give you the same valuable information as a straightforward aptitude assessment. Case studies can be too conceptual; rather than seeing how a candidate will approach the work listed in the job description, case studies ask abstract questions. The goal of asking “how many tennis balls can fit on a Boeing 757” is not to see if the candidate can guess the right answer, but to see how they approach the question and reason through their response.

But this knowledge doesn’t always serve a recruiter with the best predictor of on-the-job success.

Group interviews provide more insight – into a candidate’s teamwork, leadership, and communication, for example. Yet, in a group scenario, extroverts tend to dominate. It can be difficult to see how each candidate performs as an individual while trying to consider the group at once.

In summary, skill testing is all about understanding whether a candidate can do something or knows something. It’s about verifying their ability to go the distance with your company. Pre-employment assessments differ slightly in that they focus on predicting how a candidate will behave in certain scenarios, not what they can do. By combining questions from skills testing and pre-employment assessments, recruiters can get a more accurate picture of the candidate’s ability.

Additional Resources

For more reading, check out some of these great resources.

• https://www.mindtools.com
• https://www.techgig.com/skilltest
• https://www.3smartcubes.com
• https://www.criteriacorp.com
• https://www.smartrecruiters.com
• https://www.revelian.com