Best Readings on Talent Assessment
Subscribe to The Thoughtful Leader newsletter to discover leadership insights to elevate your team's performance.
Talent is important to any manager or organization, but assessing talent is far from easy. We can assess talent any number of ways including:
Job interviews
Performance reviews
Reference checks
Personality tests
Work simulations
I recently reviewed third-party literature on talent assessments, and here are my favorite articles on the topic along with summaries:
The Validity and Utility of Selection Methods in Personnel Psychology
This article from Schmidt and Hunter provides hard data on the most effective methods to assess talent. Work simulations (aka work sample tests) are the most effective with an r of .54. Surprisingly, structured job interviews are not that far behind with an r of .51.
The problem is few companies do structured interviews, leaving it to the interviewer’s whims to conduct interviews as they please. This results in unstructured interviews which have a poor validity (r) of .38.
Interestingly enough, peer ratings are competitive too with an r of .49. And it’s worth noting that that reference checks have a paltry r of .26. There is some signal to be gleaned from reference checks, but beware: there is a lot of noise!
As an aside, a group of scientists revisited these findings in a journal article published in December 2021. I'm not as familiar with the 2021 study as the one I originally posted. But it appears from the summary that the conclusions are the same as the original study, only the magnitudes are different.
Which CEO Characteristics and Abilities Matter?
In this article, Kaplan et. al. identifies the exact traits that lead to successful CEOs using statistical rigor. To summarize:
For PE-buyout CEOs
A 1 point change in being persistent, led to a +51% increase in the probability of success
A 1 point change in being proactive, led to a +43% increase in the probability of success
A 1 point change in upholding commitments, led to a +38% increase in the probability of success
For VC-backed CEOs
A 1 point change in being proactive, led to a +14% increase in the probability of success
A 1 point change in ability to remove under-performers, led to a +14% increase in the probability of success
A 1 point change in creative ability, led to a +12% increase in the probability of success
Each CEO was rated on a scale of 1 to 4 for 30 different characteristics by a well-regarded talent assessment firm. So a one point change would be an increase from a 3 to a 4. Success was defined by the CEO's respective PE/VC firm as either 0 (failure), 0.5 (mixed), or 1 (success); we can assume the exit multiple factored into that determination.
One of the most shocking conclusions is that a 1 point increase in teamwork actually reduces probability of success for PE and VC CEOs by -21% and -14% respectively. There's similarly shocking conclusions on how respect for others, enthusiasm, and oral communication affect outcomes, but it's more pronounced on the PE CEO side.
So you might not like that a-hole founder, but they're more likely to succeed. ;)
One more observation: the paper concludes that talent is (1) measurable and (2) predicts performance.
Are CEOs Different? Characteristics of Top Managers
This is a follow-on article from Kaplan where he goes back to the ghSMART dataset to deduce traits that explain top performing COOs and CFOs along with CEOs.
The conclusions are weaker this time primarily because the predictive variable is based on whether person exhibited success by “getting the job.” This stands in contrast with the initial paper where the predictive variable is whether the candidate “achieved a successful exit.”
Reinventing Performance Management
In this paper, Marcus Buckingham and Ashley Goodall proposes a new performance management philosophy that will 1) save time and 2) increase value by eliminating biases.
They assert that four questions better identify talent than any other question they explored:
Given what I know of this person’s performance, and if it were my money, I would award this person the highest possible compensation increase and bonus.
Given what I know of this person’s performance, I would always want him or her on my team.
This person is at risk for low performance.
This person is ready for promotion today.
Understanding the Latent Structure of Job Performance Ratings
Overall, this article concludes that performance reviews are inherently unreliable. That is, most of the difference in how an individual is rated in a performance review has nothing to do with their performance. On average, 62% of a person’s performance review largely reflects the personal quirks and “idiosyncrasies” of the rater themselves.
As you can imagine, self-appraisals are most afflicted with these quirks whereas boss ratings are less afflicted with these quirks.
