How to hire for AI skills: what the research says and what to measure instead
TL;DR
- Most organisations hire for AI skills by screening for tool proficiency. The problem is that tool proficiency is a short-term signal that tells you very little about long-term performance.
- Research on technology adoption shows that learning agility, analytical thinking, and adaptability predict success in fast-changing technical environments better than experience with specific tools.
- The underlying competencies that drive performance in AI-adjacent roles are measurable, durable, and available to assess at scale. However, this is only true if hiring teams stop chasing tool lists and start using structured assessments to measure the traits that matter.
Most organisations measure the wrong things when it comes to hiring for AI capability.
After all, knowing how to use a specific generative AI tool doesn’t mean you can learn to use a different one, think critically about outputs, or make sound judgments about when AI works best.
The problem compounds quickly. Job descriptions list AI tools as requirements, interviews test for surface familiarity, and hiring pipelines fill with candidates who have memorised prompts but struggle to evaluate results. To fix this, organisations need to ask sharper questions: What does it actually mean to have AI skills? And what predicts long-term performance in roles where those skills matter?
Why “AI skills” is not a simple thing to hire for
AI skills are not a single, stable capability. The tool landscape changes faster than any recruitment process can track, which means assessing tool proficiency is often a misleading signal.
Hiring teams tend to conflate three distinct layers of AI capability:
- Tool proficiency: This layer covers the use of specific generative AI tools, like ChatGPT, Microsoft Copilot, or Google AI. Unfortunately, this layer is highly perishable, as it varies with every version update. Anyone with hands-on experience and motivation can acquire it quickly.
- AI fluency: This layer goes a step deeper. Candidates in this category understand how AI models work, where they fail, and how to evaluate their outputs rather than accepting them at face value. This is more durable and more predictive of good decision-making.
- AI mindset: This is the deepest layer. It describes the cognitive and behavioural traits that allow someone to adapt to new tools, spot appropriate use cases, and work well in AI-augmented environments. It’s the most durable layer and the most predictive of long-term performance.
Most hiring processes only assess the first layer, which leads to concerning gaps. To succeed in the modern, AI-powered environment, your hiring team needs to assess all three layers.
Learn more about the nuances of hiring for AI skills from our e-book: Stop Hiring for AI Skills
What the research says about AI capability and performance
Organisational psychology research is consistent: The competencies that predict performance in changing technical environments aren’t always related to one’s technical expertise.
- Learning agility: This ability is one of the strongest predictors of performance in roles where the technology environment keeps shifting. People who learn quickly—and apply that learning across different contexts—are better positioned to develop proficiency with new tools as they emerge, rather than waiting for formal AI training or a new course to catch up.
- Analytical thinking and critical evaluation: These abilities are central to many roles. As AI systems generate insights and surface recommendations, the humans working alongside them need to ensure the outputs are reliable, identify errors, and make sound decisions based on incomplete or probabilistic information. It’s a cognitive skill that transfers across systems.
- Adaptability and resilience: These abilities matter because working with AI is iterative and experimental by nature. People who can navigate ambiguous situations and rapidly recover from failure are better equipped to make real progress in AI-augmented daily work. These skills prevent them from getting stuck when a workflow doesn’t behave as they expect it to.
- Ethical reasoning: This ability is a strong performance differentiator, especially as AI becomes embedded in the most consequential decisions. The ability to identify risks, flag concerns, and apply judgment about appropriate use is a practical capability. It affects the quality of outcomes in roles where AI-assisted decisions impact candidates, customers, and/or employees.
Good news: The competencies above are what Sapia.ai‘s assessment framework measures. Why? Not because they’re abstract ideals, but because they’re derived from analysis of over 37,000 job descriptions and validated against real hiring outcomes. As such, we know they’re important.
What most hiring processes get wrong when assessing AI skills
Most AI skills assessments, at least in the screening stage, rely on self-reported tool experience.
This gives technically confident candidates an easy advantage, as they can boast about their AI skills in their CVs and throughout unstructured interviews to impress hiring managers.
It can lead to other problems as well. For example, candidates who can talk about AI tools aren’t always able to make good decisions when using them. Plus, unstructured interviews favour confident, articulate communicators over candidates who demonstrate stronger underlying analytical capabilities.
At the end of the day, requiring specific tool experience excludes people who have the competency to learn quickly and would outperform surface-familiar candidates within months of joining your team.
In terms of skills-first hiring, assess for demonstrated capability and underlying potential. This approach is more predictive than screening for credentials and prior experience.
What to measure instead (and how to measure it)
If measuring AI skills is bad process, what should you measure instead? These three things…
Learning agility and adaptability
To assess these competencies, you need to ask candidates structured behavioural questions.
How have they navigated a situation in which the tools or processes they relied on changed? What did they do, how quickly did they adapt, and what did they learn from the experience?
Sapia.ai’s Chat Interview assesses learning agility and adaptability as named competencies within our 25-competency framework. As such, it scores candidate responses against validated rubrics rather than leaving the evaluations to each interviewer’s interpretation. This process removes the inconsistency that benefits candidates with more confidence than real-world capabilities.
Analytical thinking and critical evaluation
Next, craft situational questions that ask candidates to evaluate a plausible AI output.
Doing so provides direct insight into their thinking skills without requiring tool-specific knowledge. What looks right? What looks wrong? What further information would they need to act?
Situational questions are harder to answer because candidates can’t give rehearsed responses. So, answers are more predictive of real performance. Situational questions also tell employers how a candidate will function in a role where AI works as a co-pilot, not a replacement.
Ethical reasoning and risk awareness
Finally, ask candidates to reason through scenarios in which an AI-generated recommendation might carry unintended consequences. Then, assess whether they raise the right concerns.
These questions are important—especially as AI is often used in decisions that affect large amounts of people. Candidates who can identify risks and apply principled judgement about appropriate use become a significant business asset. Sussing these skills out in a structured interview context is key.
Building an AI skills hiring process that holds up as tools keep changing
As an HR or TA leader, you need to build an AI skills hiring process that works in the real world.
First, define what AI capability means for every job title before you write a job description or interview question. For example, a customer service role using AI-assisted responses has different requirements than a data analyst role tasked with building AI-powered workflows. Lumping them under a single “AI skills” banner will create huge amounts of confusion, slowing your team down in the process.
Second, use a competency framework to anchor assessment criteria to durable traits, not a tool list. That way, when tools change, you don’t have to rebuild your competency-based framework from scratch.
Third, apply a structured assessment across all candidates. Inconsistent, unstructured interviews benefit confident communicators and create compliance risks. Structured tools that score against validated rubrics give every candidate a fair opportunity to demonstrate the capabilities that predict performance.
Last but not least, build a review cadence into your AI skills hiring process. As AI and machine learning tools evolve and role requirements shift, you should revisit assessment criteria on a regular basis to ensure they’re predictive. This is an ongoing process, not a one-time setup.
Hire for responsible AI principles, not existing AI skills
Hiring for AI skills is about finding people who can learn, adapt, judge, and reason as the AI tools change. After all, you can always support AI upskilling in the future. The candidate’s mindset is most important.
To find the right candidates, your organisation needs to measure the right things. That way, you can build AI-capable teams that hold up over time. If you only screen for tool familiarity, on the other hand, you’ll hire candidates who look capable on paper but struggle in practice.
The competencies that predict performance in AI-adjacent roles are measurable. At Sapia.ai, we’ve built our platform to assess those competencies at scale, fairly, and with the proper evidence.
Book a demo to see how Sapia.ai assesses competencies and predicts performance in AI-adjacent roles.
FAQs about artificial intelligence in the workplace
AI skills in a workplace context mean the ability to work alongside AI tools. This includes knowing when AI works well and when it doesn’t, evaluating outputs with a critical eye, adapting as tools change, and applying sound judgment to AI-assisted decision making scenarios.
Tool proficiency is knowing how to operate specific tools. AI fluency is understanding how AI models work, where they fail, and how to evaluate what they produce. AI fluency is more durable and more predictive of performance, which is why you should assess it during the hiring process.
Learning agility, analytical thinking, adaptability, and ethical reasoning are the strongest predictors of long-term performance. These traits transfer across tools and environments, which is why they’re a much better hiring signal than tool-specific experience.
Learning agility predicts the speed at which someone will develop proficiency with a new AI tool, adapt to new systems, and perform daily tasks in a shifting environment. As such, learning agility measures future capability, not current configuration. Meanwhile, tool familiarity is perishable.
Use situational questions that ask candidates to evaluate AI outputs, identify risks, and reason through ambiguous scenarios. Situational questions reveal analytical thinking and ethical reasoning without favouring candidates who already know how to use certain tools.
Focus on underlying competencies rather than listing specific tools. Describe the decisions the person will make, the judgements they will need to apply, and the environments they will work in. This broadens the talent pool to include high-potential candidates, even if they don’t have specific tool knowledge.
Review criteria whenever there’s a significant shift in workflows or tooling, as they relate to certain roles. A competency-based framework reduces the frequency of major overhauls, but regular calibration keeps assessment criteria predictive.
About Author
