Contents

Background

Reference calls are tough. Most often, by the time you do them, you’re pretty sure the candidate is a good fit for the role. You’re looking for novel evidence — anything that would make you second-guess the decision to hire them, or that would change how you think about their responsibilities.

But that purpose runs counter to pretty much everyone’s desire to not speak ill of others.

Whether the reference was given to you by the candidate or it’s a backchannel, the person on the other end of the line tends to be positive. That default positivity leaves you reading the tea leaves and making often-flawed inferential leaps from their tone of voice or the way they answered the question.

So this all boils down to: how can you get the negative or constructive parts of the truth about a candidate?

Here are the two tools I’ve been happiest with:

  1. Granting implicit permission for honesty
  2. Forced quantitative questions

Tool #1: granting implicit permission for honesty

This requires you having interviewed the candidate — we’ll call them Carlos — earlier in the process and pulled out specific challenging situations or feedback from him that you can then reference. Good questions for this include:

  • what was the feedback you got most often when you were working at that job?
  • tell me about a time when you had a conflict with someone there
  • if I were to look at your performance reviews from that job, what themes would I see in terms of areas for improvement?
  • what was the worst decision you made there?

These are all pretty standard interview questions. You might get self-aware candor or you might get a watered-down answer. But either way, you now have a nugget you can use on the reference calls. In my opinion, this is the best reason to ask those questions of candidates — not because you think their direct answers are particularly useful.

When you’re on the phone with the reference — Rachel — now you can say things like:

So Rachel, Carlos was very open with me — which I really appreciated — and told me about…

  • … getting a bunch of feedback on his lack of attention to detail. Does that resonate with you?
  • … the issue he had with Doug about the timeline for the mobile app launch. Do you remember that?
  • … his decision to overrule his team on hiring someone. Do you remember how that went down?

Once you say this, Rachel will very often open right up. She’ll feel the implicit permission to share her perspective on the situation from Carlos, since now she knows Carlos was already transparent with you about it.

Ideally, you can keep pushing down the rabbithole, showing more of your knowledge about it (which you got from Carlos):

That’s very consistent with what he told me. It sounded like it might have caused a bit of a lingering issue, too… is that how you felt?

People rarely want to break the news to you of a bad decision or a category of critical feedback. But once you’ve shown that the candidate themselves already broke it — and that you have an understanding of it — they are very often willing to fill in the gaps in a much more meaningful way.

This is, in my experience, by far the most effective tool on reference calls: grant the respondent implicit permission from the candidate to answer honestly. And the simple way to do that:

  1. get a nugget about a negative situation or decision or critical feedback from the candidate
  2. tell the reference that the candidate told you about that already
  3. and ask their perspective and dig in deeper

Tool #2: forced quantitative questions

People can come up with all sorts of innuendos, scapegoat answers, and ways to avoid the question when you ask them typical prompts (“what were Carlos’s biggest areas for improvement?”).

So one way to get the truth is to force a quantitative response with a well-structured question. Sometimes this is easy because there are quantitative individual results to measure (“after he ramped, was Carlos in the top 25% of salespeople by quota attainment? Top 10%?”) — but if that’s the case, you’ve probably already gotten that information from the candidate and are simply verifying it.

So how can you get new information in a quantitative format? I’ve found the best way is to transform a qualitative question into a quantitive one. Here are four questions I love (the first two are great direct interview questions too, not just reference call questions):

Scored skills

An example for an engineering manager you’re referencing:

In this role, there are four skills we really care about: technical depth, recruiting/hiring, team management and leadership, and broader communication.

If I gave you forty “points” and you had to allocate those for Carlos based on how good he is at each, how would you allocate them? That is, if you think he’s equally good at all four, he’d get 10 points each — but I imagine he’s likely better at some than others.

I love this question (again: great direct interview question too). It forces people to make tradeoffs and expose what someone is really good at — and what they’re less good at.

A typical answer comes after some clarification and back-and-forth — maybe it’s:

So for Carlos… maybe a 15 for technical depth, 10 for team management and leadership, and then… let’s say 9 for communication and 6 for recruiting. I feel bad only giving him a six there but I guess I need to make some tradeoffs!

This, of course, gives you more to dig in on with the reference or with the candidate.

Ranked attributes

This is somewhat similar to the “scored skills” question. I picked it up from What’s going on here, with this human? (an excellent writeup from Graham Duncan on interviewing and referencing):

Three attributes I like to keep in mind are someone’s hunger, their humility, and how smart they are about people. If you were to force rank those for Jane from what she exhibits the most to least, how would you rank them?

I usually ask it slightly differently:

To succeed at this company / in this role, someone would need hunger, humility, and smarts about people. I think Carlos has all three — although tell me if you disagree! But, of course, everyone has a different balance of those attributes. So if you had to rank those three, from what Carlos exhibits most to least, how would you rank them?

Endorsement strength

Another one from What’s going on here, with this human?!

How strong is your endorsement of Jane on a 1-10? (If they answer 7, say actually sorry 7s are not allowed, 6 or 8? If the answer is an 8, “What is in that two points?”)

I think this question is great. But I’d inquire further no matter what they say:

6 or below (rarely said): “what would it have taken for them to make it to [number plus two]?”

7: “Sorry, 7s are not allowed. If forced to choose, would you go with 6 or 8?”

8: “Why an 8 not a 6? what is in that two points?”

9: “that’s amazing. I’m not surprised. What would it take for them to get to 10?”

If they say “oh I just wouldn’t give a 10” or something similar (they usually do), I say:

“Hah, yep, I’m the same way. It would take something superhuman to hit a 10. For Carlos in his role… what would that have been?”

10: “incredible, although I’m not surprised — Carlos is clearly exceptional. I need to ask, though — would you ever give someone an 11?”

Then:

Rachel: “yes, I didn’t realize that was possible — it would be Carlos”

You: “that’s great.”

or

Rachel: “yes, I guess I could”

You: “for Carlos to have hit that superhuman 11… what would that have taken?”

or

Rachel: “no, 10 is top of the line.”

You: “does anyone else in a similar role get a 10 for you? If so I’d love to know how they and Carlos are different despite both being 10s.”

Top percentile

“Of all the engineering managers [or “engineers”, or “managers”, or whatever] that you’ve worked with in some capacity, where does Carlos fall percentile-wise? As in, top 50% of all engineering managers? Top 25%? Top 10%? Or what?”

People will usually say “top 10%” (if they don’t, then you’ve collected very useful information!). Once they say top 10%, say:

That’s great. I’m not surprised. What would it have taken Carlos to be in the top 1% you’ve ever worked with?

Sometimes they will say “oh wow, I didn’t realize top 1% was an option. He’s definitely there.” Occasionally you get a stickler who says “well they couldn’t be top 1% because I haven’t worked with that many people in that role.” If they say that, then just simply ask “fair enough — are they the best you’ve ever worked with in that role?”

But more often, you’ll hear something like, “oh, I mean he was really great. But to be top 1%… I guess maybe just been a little more communicative.” And now you have a piece of information you can start digging deeper on.

Again: this question is all about forcing a quantification of a qualitative judgement, and works particularly well by not explicitly giving the respondent the option of the highest rating.

If Carlos was really top 1%, then they’ll say “oh more than top 10%… he’s the best ever.” And if not, they can still feel good about picking the highest choice presented (top 10%), but then when you follow up, they feel obligated to answer honestly about what would have made the candidate even better.

What next?

I’ll make another post later — and link it here — about other aspects of reference calls. How to reach out, backchannel vs. not, who to talk to, how to talk to the candidate about the results of the calls, the process on the call itself, etc.

But for now, I hope these tools for finding the truth are useful.

Looking for more to read?

Or, more on reference calls:

Who, by Geoff Smart and Randy Street, is a great read on hiring, and has lots of good bits on references. Thanks to Matt Mochary for the recommendation many years ago.

I mentioned this earlier in the post: Graham Duncan’s article What’s going on here, with this human? is excellent.

And for even more...

Want to hear about new essays? Subscribe to my roughly-monthly newsletter recapping my recent writing and things I'm enjoying:

And I'd love to hear from you directly: andy@andybromberg.com