John Vlastelica: [00:00:00] I think when, when Joel and Rick Dalzell our CIO and I, those are the three kind of creators of the program. I think when we were building out the program, we were saying, you know, Hey, there's an opportunity here to kind of embed a very high hiring bar. And, and the way to embed that into all these decisions is to identify in the beginning.
It was 20 people in the technology organization that were not trained up to be bar raisers. They were bar raisers already.
Dave Schappell: [00:00:35] This episode is sponsored by recruiting toolbox, founded by ex Amazonian, John Velasquez. Recruiting toolbox is a training and consulting firm hired by world-class companies to raise the bar on how they hire. They've trained over 10,000 hiring managers, interviewers and recruiters from 30 countries to learn more, visit recruiting, toolbox.com.
Today. I'm extremely excited to be talking with John , who was Amazon's first technical recruiting manager hired in 1998. And the reason for my excitement and this is true excitement is that by far the most common feedback I've had from podcast listeners is mom close your ears. How in the hell did you hire that?
Amazon hire all of those amazing people that we've interviewed thus far and well, Joe, John's going to tell you, cause he developed many of Amazon's scalable early recruiting processes, including the bar raisers program. Amazon's interview training program called making great hiring decisions and more, and both of those are still in place today, which is pretty crazy given that it's been 20 years, uh, John's going to share stories about why those programs were necessary at the time and what was the key to their success.
And I truly believe that every startup and most large companies too, uh, will benefit from hearing these stories from John. So welcome, John. 
John Vlastelica: [00:01:52] Yeah. Thank you, Dave. I'm excited to be here. 
Dave Schappell: [00:01:54] Great. So it's hard, hard to talk about Amazon's recruiting process. Without someone mentioning the bar raisers program, whether it's in, you know, verbally or in print, can you tell listeners what it is and probably more important why it was created? What, why it was necessary? 
John Vlastelica: [00:02:12] Yeah. So the bar is, or program was, was designed to help maintain and scale a high hiring bar across Amazon. We, we built the program in 1999, as we were getting ready to go through some massive company growth. And the bar raiser is a, is a role. It's a person who sits in on the interview team, who is not part of the, you know, is not the hiring manager is not the recruiter.
And they, their job is primarily to make sure that we maintain a high hiring bar higher for the longterm. Make sure it's a good quality hire for the company. Their incentives are not to put a button. The seat, their incentives are not to please the hiring manager, but to make sure that we're hiring someone who is good for the company.
And as part of that, they are given veto. Which is something we'll probably explore a little bit because it's, it's, it's an interesting thing to add to someone's responsibility. I think many companies will regularly invite someone to their interview team that maybe sits outside the team. So maybe Dave, if you were interviewing for a job, you might have your boss be part of the interview process and they would kind of, you know, rubber stamp it or say, I agree, or whatever, give you some feedback.
But this was someone who wasn't necessarily, well, certainly wasn't the hiring manager or the recruiter, and wasn't necessarily even part of the team. And so they came in and made sure that we made a great hire for the company. 
Dave Schappell: [00:03:26] And, uh, there's one thing you just mentioned there that I really like is that it not only can't be the horror is, or can't be somebody sort of on their own team or in that org or the hiring manager.
It also can't be the recruiter because both of those people could have sort of perverse incentives, right. To hire somebody because they're under extreme pressure or that recruiter could have a goal to hit. They need to hire 30 people this quarter or something like that. Is that, is that sort of agreement.
John Vlastelica: [00:03:51] Yeah. I mean to some degree. Yeah. But it's also designed to make sure, sure. That you are hiring a good Amazonia, not, not just hiring someone that meets the specific wrecks. You know, I don't know about you Dave, but I had seven bosses and six years at Amazon. Um, it probably equal number of different office buildings.
I was in different groups and you know, you want to make sure that we're not just hiring. If Dave, if you're the hiring manager, you're hiring someone that fits your kind of affinity bias preferences that are hiring, it's a good hire for Dave and needs to be a good hire for the company. And so a lot of it was really designed to make sure that the kind of person we were hiring based on the bar raisers kind of input as well was the kind of person that could survive a reorg.
You know, we didn't go like crap. We just changed deals. And now we need to move this person over here, but they're not a good hire for any other team. Right. It's very common in companies to give a lot of ownership to the hiring manager, to. Whoever they want. And they start with their buddies and their friends and all these biases kick in.
And then if that leader leaves, what happens to that team, right? 
Dave Schappell: [00:04:46] It's just that, it's, it's the way that you start getting weak areas of your organization, right? Like you can have, you can have a killer, you know, I don't know, order pipeline group, but then you hire a manager who maybe was a mistake and a B or a C player.
And they start hiring people that they used to manage in their past company. And then those are those people hire poor people and you end up this, you know, this problem entire area, not just the person, that person can sort of spawn a problem, uh, that, that the org can have to deal with for years. 
John Vlastelica: [00:05:16] And you just captured a big, you know, a big part of the origin story.
The why for bar raisers, as, as we were scaling up, I started in spring of 1998. And by 1999, you know, we were hiring, I think in one month in 99, we hired the same number of people we hired in all of 1998, for example. So, I mean, we've really scaled up and you had new people hiring new people, hiring new.
Yeah. And what happens if that new person who's brought in as a director or a manager, and they're a hiring manager, building a team is not awesome and we don't catch it. You know, we have a false, positive hiring decision and they turned out to be great. And now they start building their team out. Even if you deal with that manager later, now you've got their team and it doesn't mean that all bad managers hire only bad people or something like that, but there's a risk there.
And so that's really what, what bar raisers, the origin story was really around trying to prevent bad hires. It's really focused on avoiding the false positive hiring decision. 
Dave Schappell: [00:06:09] It's in a company. That's, I mean, this is most critical, this topic and this best practices, most critical for fast growing companies.
Like if you're at a large company, that's really not hiring that many people. You can't do much damage, but in a lot of these really small startups, they'll go from 10 people to 40 people in a year because they raised venture capital and they hire right. And you can, you can totally lose your. Like overnight, like, you know, I, I, I just, I did, I just read, um, the chapter in working backwards, uh, Colin and Bill's book, uh, that I think you wrote or helped write.
And it just, it talked about that that really. 
John Vlastelica: [00:06:44] Yeah. Yeah. Colin, Colin and bill wrote the chapter, but I got to provide some input, but that's exactly it. I mean, there is so much risk around hiring, you know, if I were, if I were coaching an entrepreneur or a startup today, you know, you want to over resource, you know, you're, you're hiring, you want, you want to put a bigger investment into it than you think you need.
Uh, and, and bar raisers for us was a mechanism, not just a quality control mechanism and kind of an inspection mechanism, but it was also one of the ways that we, we deployed a high bar, right bar raisers. We're not just there to say you, you have my approval or not to hire this person. They were really there as a coach and a mentor.
They were designed to. My opinion, almost work themselves out of a job over time to get to the point where everyone is. Part of the interview process is learning from this well-respected high hiring standards bar raiser, how to interview effectively. And so borrower users were asked to not only come in with a strong opinion, backed by evidence, but to help facilitate the hiring decision meeting itself and to help make the rest of the interviewers better at their craft, you know, help them be better.
Interviewers feedback on feedback for example, was something. Yeah. You don't see that a very many companies. And I've worked with hundreds of companies since I left Amazon as a consultant. And I'll tell you, you rarely get feedback on your feedback. You rarely get a post-mortem on that hire who is amazing, or it didn't work out.
And Amazon's culture was so much around, you know, what can we learn from this? How do we, how do we scale? And, you know, have an inspection mechanism and dissect where there were problems or find out what worked really well and how do we elevate and replicate that and make that the new kind of cultural norm, the process norm, and some bar raisers were a phenomenal mechanism.
Dave Schappell: [00:08:17] Yeah. I told you before, like I was a terrible interviewer my first year or two, and it was an uncomfortable process. And so I was probably 2001. I said it as one of my goals for the year. Was to become a better interviewer. And so I, you know, shadowed a bar raiser and I eventually became a bar raiser. But what the ironic part is when you put that time in interviewing went from being stressful to being kind of, you know, you had a, you had a formula for how to do it.
You knew what to do. So it wasn't as stressful for me anymore. Cause you sort of knew how to develop a, uh, an informed opinion and document it. And then, uh, if you, if you were confused, you actually could talk to the bar raiser at the, at the debrief. And you'd say, you know what? I feel like I flubbed this part of the interview and it was actually, you know, it was just, it was, it was, I thought it was always an example of a really great ownership, like a display of ownership.
It was making every recruiter or every person who's interviewing kind of act like they're the owner of the company. And they're trying to make a really good hiring decision decision to increase the performance of the team and the company as whole. 
John Vlastelica: [00:09:18] Absolutely. Yeah. And, you know, Jeff Bezos in his letter to shareholders, the one that was probably most impactful for me was his 1998 shareholder letter, which came out in early 99.
And, you know, he talks a lot about, um, asking ourselves key questions in those hiring meetings or decision meetings. And, and one of those questions is how does this person make our overall team better, right? How are they going to improve the effectiveness of the overall team? And I think that's something that, again, the bar raiser has much more of that owner mentality looking long-term beyond the short term, I got to get a button to seed.
I need to hire a software engineer by Tuesday or else we're screwed. You know, this, this person is saying, you know, let's, let's not let the, the speed pressure, you know, their speed and quality tension in, in many hiring processes. And so they were much more focused on, on the quality piece yeah. 
Dave Schappell: [00:10:05] Or questioned about the, I almost skipped over the part.
So who created the, uh, were you hired to create this? Was it, or was it just something that became obvious was necessarily. People that created the program. And I know we have some of the origin documents that we can post, which is kind of cool. Um, and then talk about like, how were the first, how many original bar raisers were there?
Like how were they picked that sort of thing? Cause now there's probably thousands. Right? Um, but in the early days it must've been a smoker. 
John Vlastelica: [00:10:33] Yeah. So I was hired, uh, as we called it recruiting STG for strategic growth. So I was hired as a manager of strategic growth to look after technology hiring. And, uh, I was not hired to build bar raisers.
I never heard of oranges or even that concept, uh, especially, but Joel Spiegel, who you interviewed was our VP of engineering. And my recollection is that Joel, uh, after having an experience at Microsoft, which had a program called the as appropriate program, which basically put a final interviewer who also had veto power, uh, who had to kind of approve the hire, not, not someone from necessarily outside the department, it might've been a functional VP or something.
Um, I think when, when Joel and Rick Dalzell our CIO and I, those are the three kind of creators of the program. I think when we were building out the program, we were saying, you know, Hey, there's an opportunity here to kind of embed a very high hiring bar. And, and the way to embed that into all these decisions is to identify in the beginning.
It was 20 people in the technology organization that were not trained up to be bar raisers. They were bar raisers already. We were just kind of acknowledging or formalizing something that they were already doing. And let's initially we called them bar keepers and then it changed to bar raisers. And I cannot remember it was so long ago, day, but yeah.
I remember we were thinking, we don't want to keep the bar. We want to raise the bar. So I don't remember who said it or how it changed, but, uh, but, but the idea was to embed this high hiring standard into every interview process, which by the way, created an incredible constraint because these 20 people who were selected were some of the busiest, most respected, most effective, biggest workloads.
Dave Schappell: [00:12:10] Well, they were also, and they were all engineers. Right. And then an engineering story. Cause it started in tech. Yeah. How long did it take to move from technology then to other parts of the org and how did it change and how did the change when it shifted? 
John Vlastelica: [00:12:26] Yeah. And you know, it's interesting. It, it was, it was, we kind of, it was forced because we decentralized as technology.
So technology has to all be, you know, all the front end back end. All the systems support, all lived inside of a central organization under Rick dolls, L uh, and then as the technology organization became decentralized. So the teams that supported supply chain or operations moved into operations and eventually reported up to an operations leader.
You had these tech teams that were employing a different process with this bar raiser than the non-tech teams. And of course the engineering teams were. Sometimes invited to interview maybe for a product manager or, you know, some kind of supply chain expert. And there was a, you know, why aren't we using bar raisers?
And so I can't say that we like had a very programmatic, like on this date, we're going to launch bar raisers across the whole company, but it became pretty obvious that other teams had not had that kind of same discipline. Um, didn't have someone that was looking at the kind of things a bar raiser looks at, um, as a, as a discrete role.
I mean, certainly the company was good at hiring people outside of tech. Of course. I mean, I'm not suggesting they had a lower bar or anything like that outside of engineering, but there wasn't the same kind of mechanism in place. And in some, in some ways there had been a little bit of a looseness. So Jeff Bezos personally brought this idea of, you know, a hiring meeting structures.
I credit Jeff with kind of bringing to early Amazon, um, as kind of the Amazon way. And we would have, you know, big packets with reference checks, we would read and meetings and, you know, written feedback as you remember, Dave. I have, I have seen only one or two companies in my entire career now that have the discipline around written interview feedback that Amazon has, it is absolutely insane and way beyond, you know, a paragraph, uh, or, or a thumbs up in a, in a system.
So you had this, this kind of deployment out and along with the, the bar raisers as it started to, to kind of spread it out to other teams. And I think the timing was, you know, 2000 to 2000. Ish, you know, plus or minus, uh, you started to have a lot of the other kind of discipline around some of the other areas kind of either re-emerge or get more serious because you had a bar raiser, demanding, written feedback that looked more than just a thumbs up or thumbs down.
Dave Schappell: [00:14:34] I think I told you before, like the one thing that I really like about the bar raiser program, and then the bar raiser, the person is that they manage the hiring meeting or the, you know, the, the review of the interview. And that's, that's a good thing. Like, because let's just say you are someone who writes really detailed feedback.
And then two of the other people don't, you're the jerk. If you mentioned it to them, like, Hey, I couldn't make it, you know, but the bar is, or it's their responsibility to do it. And I've been on the receiving end of that too, where I was just so busy and it was incomplete. And I like that I'm being called on it because I sorta know that the other people are going to get called on it.
That's what makes the process better for everybody longterm? Um, yeah, I mean, yeah. 
John Vlastelica: [00:15:13] And write writing and writing or feedback. Dave, I don't know if you've had this experience. I certainly did where I come out of an interview and I took notes on the interview, but then I had to sit down and actually write out like justify, rationalize my hire, no hire.
And I started writing with maybe this kind of inclination, the more I wrote, I went, what am I doing? This is this person. You know? And so, uh, as a bar or myself also for the HR, our HR function, I found myself really appreciating how just the, the sitting down to kind of collect your thoughts. It took some time.
I mean, I will spend 15, 20 minutes writing feedback, but it forced you to kind of, you forced higher decision quality, cause it forced you to better articulate your reasons. Why based on evidence, not based on biases. 
Dave Schappell: [00:15:54] And we're going to get to that in the next question, because I, cause it actually, I'm going to, I'll say something that builds on that.
I like it when, cause you go, you do, you sit down and you start to write and it becomes clear that your initial impression wasn't supported by the facts or you're starting to develop some misgivings here, but, and then, so you generally, that means not inclined like in your, you start to have that feeling it's not inclined.
And then you get into the debrief and it's the best feeling in the world when for other people also pick up on other things. And whereas if it were only you doing that interview, you might have, you know, in a rush been inclined to just give the person a chance. But anyway, so, so following on bar raisers and very tied to it, I th you know, following the next big program or you've probably a lot of them, but the one we'll talk about, uh, was called was Amazon's interview training program called making great hiring decisions. Um, can you talk about what that was? And then also, why did you call it. 
John Vlastelica: [00:16:49] Yeah, so making great hiring decisions, was something I built. I think it was 2001 that was, designed to take some of the bar raising practices and things that I'd learned I came into Amazon I thought I was a good interviewer I'd taken behavioral interview training before I knew the legal do's and don'ts I hired engineers at my last company. And, I was, Joel Spiegel and other people that interviewed me, saw something in me, but I wasn't nearly as good of an interviewer as people like Joel, for example, or Jeff Holden or JJ Jacobior or other people in the engineering team that are leaders.
And so I found myself learning a lot from some of the people you've been interviewing and realize we should really capture this in a training and begin to push it out and be a little more prescriptive. So we've already identified these bar raisers. We know they have really good judgment. How do we take what, they do and try and bake it in.
I wanted to call the training, making great hiring decisions, because I think a lot of training that had seen that comes out of HR is too focused on kind of the, inputs of how do you ask a behavioral interview question and doesn't focus enough on the interview is a means to an end, it's ultimately to hire great talent for your organization.
It's not just to learn about the person or, demonstrate curiosity or to, make someone sweat or something at a tough interview. we're really trying to make sure we're making quality decisions and quality decisions doesn't necessarily mean saying yes to everybody. Obviously there's a lot of no's in there.
And so, I wanted to make sure that the focus was on the actual hiring decision part. Um, one of the things that I really miss about Amazon, if I'm honest, uh, is this, this decision inequality at that company is just unbelievable. The quality of decision-making the, the rigor that goes making a decision is just unreal.
And for me, bar raisers represented that really high decision quality as facilitators as one of the inputs. But also, as you said, running the hiring meeting or the debrief meeting to make sure we're making a quality decision. And I think, you know, initially when we built making great hiring decisions, when we rolled it out, it was very focused on avoiding bad hires.
I think if you fast forward to, you know, the 2010s and today it's also like, how do we make sure we're not missing out on good hires? How do we avoid false negative hiring decisions, especially with the diversity lens. Initially, I think it was a three hour program, you know, it was, it was very prescriptive.
Uh, we didn't have our leadership principles in 2001 when I built this. So we were using a, a competency model that was based some, you know, largely on the values of the company at the time, but had a few other things in there. Uh, that were important to use as kind of predictors of on the job success. So we had some, some things like ownership, you would expect to see that we were evaluating on.
Cause that was definitely a core value, maybe the most important value, um, next to kind of customer focused customer obsession. Right. Uh, so that's kind of the origin of that program. 
Dave Schappell: [00:19:28] Well, in one, so I'll, I'll first compliment you because it's, uh, a lot of human resources training is terrible, right? It's like compliance training or you're, you know, this is, this is a good example though, of something that I learned at Amazon and I've taken it everywhere and I've shared it with other people.
And I think a lot of Amazonians would say the same thing because interviewing is hard. If you don't have a plan going in. So you mentioned, I kind of described making great hiring decisions. Usually two words is behavioral interviewing. Uh, and you mentioned it, can you, for people that aren't aware of what it is, and then maybe just give an example of how you would ask a question versus how someone.
You know, versus a manhole cover question, you know, type of thing. 
John Vlastelica: [00:20:10] A brain teaser or something. Yeah, sure. So the premise of behavioral interviewing questions are, past performance is a good predictor of future performance. So I'm gonna ask you a question, Dave, tell me about a time you had to blank or blank is something that's related to the job.
if you say customer obsession is what I'm looking for, you would ask something like that. Yeah. I might ask you a question like, Dave, tell me about a time you built a product and you did an early iteration and you gave it to the customers. And talk to me about the kind of feedback you received and what kind of tweaks you made as a result of the feedback.
And then I would kind of dig and probe to understand your thinking around that and how you weighted some of the criteria and what you ultimately built, and if it was successful and all this kind of follow on probing questions, but the idea is to get a real example, as opposed to saying Dave, what's your philosophy on customers or Dave named three companies that you think are incredibly customer focused and describe them to me.
as opposed to that kind of high-level academic stuff, it's really forcing you to give a specific example and it's harder to bullshit. If someone is really pushing you for details from your past, you may have some examples prepared, but when they go beyond the what and the why, and they start digging into the, how, how did you do that?
It's much harder to kind of BS your way through that. Joel. So behavioral interviewing is pretty effective. 
Dave Schappell: [00:21:22] Yeah. Joel taught me a lot of this stuff too. And he was just like, he's like peel back the onion. Like you start, you start with the first answer. And then, you know, you want to say, well, what was your role in solving that problem?
Cause it sounds like you, it does sound like you weren't the manager, you were the underlying. So what did you bring? What would have happened if you weren't part of that team and you just keep peeling back until the other, there's nothing else there, or you're totally confident that this person was the star of that team, even if they're not saying it.
Um, so you just keep peeling back the onion until you feel confident, you know, Uh, a good take on that leadership principle. It wasn't that then, but that, that, um, uh, yeah. 
John Vlastelica: [00:21:57] Absolutely. And Joel, Joel was my scariest interview. By the way, he interviewed me for my job. I met him at a Starbucks out in Redmond, near Microsoft headquarters.
And, uh, he was a hardcore interviewer. I mean, he was by far my, my toughest interview. And so when I got the offer, I was kind of like, I was kind of wondering that I passed the Joel. 
Dave Schappell: [00:22:14] Was he asking those types of questions then? Or was behavioral interview. 
John Vlastelica: [00:22:20] Be honest, I'm not trying to diss anybody else that was on the interview team, but a lot of the other interviews were, were good and kind of normal traditional interview questions.
Joel was going, you know, he was digging deep and forcing me to go way beyond canned responses I had for the strengths and weaknesses kind of line of questioning or, you know, the generally walk me through your resume. And tell me about some of the stuff you did. He was, he was, uh, he was digging for sure.
Dave Schappell: [00:22:41] Well, for the listener. This might be the third time now. So I promise I won't do another one, but like the truth is when you hear that type of questioning, it is easier as an interviewer because you have a plan. He was like, you're trying to evaluate ownership and customer obsession and you, um, but I actually think it's easier for the interviewee as well, you know, versus I'm not saying I can't answer brain teasers, but it's just like, why are you asking me this?
Whereas when you start asking me about things and, you know, assuming I'm an expert on those things, it's a more engaging interview because you're talking about the details and diving in and, you know, and sort of validating the work that you're claiming you did. 
John Vlastelica: [00:23:18] Absolutely. You know, people that are not bullshitting that actually have done some great things.
It gives them a chance to shine. It gives them a chance to walk you through some things they did. And it, you know, again, interviewing as a means to an end, I'm trying to not answer the question. You know, what's Dave successful working at that other company. I'm trying to answer the question. Will Dave be successful working here at Amazon?
And so I need to get into the details and understand not just what you did, but how you did it and you may have done it differently. You know, you worked at a different scale company or you're working in a different geography or the problems you were facing were at different technical complexities or, or whatever.
Um, and then I asked to be good as an interviewer at being able to translate what you did there to see is do I think Dave could pick up what he learned from that and bring that to Amazon and still be effective here. But a lot of, a lot of behavioral interviewing is still, I think, too. Too general, you know, tell me about a time you had a conflict with a team member and it would be so much better to say, Dave, you work in product.
There's a lot of tension between product and engineering. Tell me about a conflict you had specifically with an engineer where you had to, and so people that are good at taking kind of generic behavioral questions and then tailoring them even more to the job. The beauty of that is not only I, as an interviewer, learn more about you Dave, as the candidate, but now you, as the candidate also learn more about the job because you know, the questions I'm asking you are communicating what's important to us.
And so it was the leadership principles became the focus and moved away from the kind of competency-based stuff that I did in kind of a first iteration based on values. As the leadership principles took over as a candidate, you can tell this is a company that walks the talk that lives, these principles, because I have a VP and a director and a recruiter or whatever, asking me a lot of detailed questions about this stuff.
I know as a candidate. You're putting a lot of weight into this. It's not just posters on the wall or something up on your career site. 
Dave Schappell: [00:25:01] And you may have said this already, but was, did making great hiring decisions come because you were getting feedback from bar raisers, that there were too many terrible interviewers or just that you, or maybe was it more that we just had to scale our base of interviewers and bar raisers longterm.
And we, we knew that there was a competency problem. 
John Vlastelica: [00:25:20] You know, I think every company, including Amazon, I'm sure there was some, you know, one HR leader in one location built some basic interview training. Um, what we wanted was a company-wide and so Amazon did not have money which have a learning and development function.
Um, you know, very limited resources. Recruiting was much larger than the rest of the HR organization. Uh, when I had an opportunity when recruiting slowed to kind of join another team and focus more on learning and development, it became obvious to me that we should start with something I know a lot about because I'm not a guru on harassment training or performance management training or something, you know, more, more broad.
So I think part of the reason we started there. Honestly, it was because I had some expertise in that area, but more importantly, yeah. Even though we weren't doing a lot of hiring in 2001, not compared to, you know, 1999 or 2003 and beyond, uh, absolutely needed to be thinking about how do we build future interviewer capabilities and capacity, right.
Because one of the, one of the real big negatives of having a bar raiser program, no surprise is especially in the beginning when we had 20, we'll have a plan to scale, it was, we absolutely would not let a, we couldn't build an interview team. We couldn't have a candidate come in unless there was a bar raiser available.
So we had to start thinking about how do you scale interviewing capabilities. We needed to take a lot of people that had never interviewed before, maybe in their life or first time hiring managers, first time people managers, and teach them the Amazon way, not just teach them behavioral interviewing.
I want to make that differentiation, there's kind of core generic training that exists. You can teach someone behavioral interviewing in 15 minutes. We want to teach them the Amazon way and embed kind of our hiring principals and the bar raiser, not bar raiser as a role, but bar raising. What does that mean?
What does that look like? When we talk about, back then, we would never, use the term culture fit today. We talked about culture and diversity differently. Back then culture fit was used at a lot of companies to define the kind of soft skills. And we said, let's not use language like that.
Let's get specific. What do we actually mean? And that's when you saw smart and customer focus and ownership. And some of these other things kind of come out. 
Dave Schappell: [00:27:20] Well, like I said, I just read the chapter in working backwards and the thing that sends it up for people that haven't been around. High growth companies.
Uh, it's it's at the end of page 33, I can barely read without glasses anymore. Um, it says, uh, good intentions. We're not going to solve the hiring problem at a company growing as rapidly as Amazon. And there you could insert Google, Facebook, Microsoft, or whatever, um, which jumped from roughly 600 employees in 1997 to 9,000 in 2000 and a hundred thousand in 2013.
And now we know it's 1.3 million or something like that. Right? So I feel it's a good cap on what we're talking about. Like, if you don't have these, some version of this at a, at a company like that, your culture, isn't what you started with. It's basically a mashup of what you hired. Um, and so if you're not trying to find people that fit your culture and I, I know culture is not the term, but fit your leadership principles in the terminology, then you shouldn't be surprised that you're.
You know, the things that your company is, uh, the, the things that people are displaying and act how they're acting, that they're different than what you expect, because you weren't looking for it when you were hiring them. 
John Vlastelica: [00:28:29] Totally. And, and, you know, we, we do a lot of, I do consulting work. Now we do a lot of interview training for companies.
And one of the things that I think about is, you know, there there's the training aspect, but then before that day of, to your point, there's kind of have we defined what good looks like. Have we aligned everyone on what good looks like. And is there kind of a federal or company wide point of view on some of these or are we letting each department or each hiring manager bring their personal preferences from where they worked in the past?
Right. You know, I use this as a, as an example, the highlight, but you know, some companies are very comfortable hiring a brilliant ass, right. Others would say, I don't care how smart you are. No assholes like that. That's just part of the deal. And so I think having a point of view, a strong point of view, um, and then making sure you get people aligned on that, that that's how you scale a high hiring bar.
Yeah, I want to bring in bar raisers. I'm like, well, good for you. But you first got to define the bar before you can raise the bar, you know? And, and we were very conscious, no surprise, Jeff Bezos. Um, and, and a lot of the leaders, you know, you couldn't get away at Amazon with just making general comments.
You had to back everything up, right? You were constantly being tested and, and people were poking holes. And so when you, if you can picture a hiring meeting or a debrief, uh, 30 minutes where, I mean, it is rigorous. The bar raiser is interviewing you, Dave, on your feedback to test. If you did your job, you were assigned these three areas.
Did you do it? You don't just get to say I'm a thumbs up, you know, and let's kind of move on. No, no, no razors basically get an interview on your feeling. I say everyone. 
Dave Schappell: [00:29:52] Supposed to, yes. For invent and simplify. Tell me what question you asked and the, and the detailed answer you, the person told you document why you're a thumbs up on invent, invent and simplify and.
Have the questions or any documented answers, it's proven that you wasted everybody's time participating in that interview. And, but, and this might sound harsh to non-Amazon listeners, but it's a, it's a re it's a re petitive process that makes everything better. You know what I mean? Like you do this a million times over people get better at it and the process reinforces itself.
Um, so yeah, you drink, you drink. 
John Vlastelica: [00:30:32] Yeah, I'm a thousand times better at a lot of things as a result of my six years at Amazon. Uh, but decision quality, as I said earlier, which was carried over into hiring decisions was just unlike anything I'd ever seen. And I've, I've, you know, a hundred examples. We didn't have time to talk about individual hiring meetings.
I recall with, you know, some of the people we've mentioned in your interviewing where I just thought, my God, like no one caught that, but them or no one dug into that, but them and how do we have they not done that? We probably would've made a hiring mistake, you know, because I remember a program manager that, uh, that Joel interviewed and, you know, we all came into the debrief, the hiring meeting thumbs up and Joel was the thumbs down.
And, you know, it was like, God, this guy seemed amazing. And I remember even telling Joel, like I would have totally hired this person for another group if I were the hiring manager. And I hear all the other feedback and Joel talked about, yeah. Did anyone ask him, you know, about the project he's working on now, this guy worked at Microsoft and you know yeah, sure.
We asked him about it, whatever it is. You know, will this program managers product or the thing that they're launching is not actually launching for another, I don't remember four or five months from now, but he's interviewing today and I'm like, yeah. Like, there is no way in hell. I would hire a product or program manager who has a project they've been working on for 18 months, that launches in four or five months.
This is going to quit before that ships, are you kidding me? Like did no one else dig? And I'm just like, oh my God. Like, but it really captured that ownership and the customer focus and the seeing things through and the, you know, it was just, it was such a powerful example. It stuck with me now 20, some years later, I think about that, about how one really good interviewer can just capture something that the rest of us just didn't even dig into.
Dave Schappell: [00:32:06] It's a side topic that I don't want to go off on because I think we could spend a half hour on it, but we all have stories of those interviewers that I'm always like if mark Amer's on an interview loop, nobody else has to be on the loop. He, he's not only hilarious, but he's the best interview I've ever been around.
Peter Cohen passed away was similarly great. Uh, Jeff Holden, you mentioned before, like there are these certain people that are so good. The nice thing is that those people, most, or all of them are bar raisers. And so they're trying to hire, you know, train and build more of those versus just bottling that up.
But anyway, um, so I have two more questions on this that I wanted to ask. One was sort of about how Amazon got away from pedigree interviewing and maybe you've, you can explain what that is versus sort of, um, attribute it's maybe, maybe beating this horse that this topic, uh, you know, uh, to the nth degree, but then also I wanted to ask about how Amazon goes about evaluating like evidence.
Like I understand it for engineers, I was always jealous of engineers cause they could ask the person to describe an algorithm or put, you know, write code on the board, but that's harder to do with a product role or, you know, maybe a content role. 
John Vlastelica: [00:33:17] Yeah, for sure. So pedigree, you know, one of the that's problematic and a lot of companies is people want to hire smart folks and they depend on proxies.
And so they will look at, you know, what school you went to they'll look at your grades, they'll look at what company you went to. Uh, they might look at your title even years of experience and there's actual science. Uh, some of it's dated, you know, early two thousands and even into the eighties, but that says, you know, the validity coefficient, the kind of predictive power of some of these things that have traditionally been used to make employment decisions are really bad, very was very low.
Um, and yet people put a lot of weight into these things, right. Uh, and so, you know, pedigree, orientation, I think that, you know, there was in the early Amazon days, there was, you know, I was asked for my sat scores, my high school and university GPA, I just taken my GMATs to get into MBA school. Um, and so someone asked me about that and I remember thinking, you know, I was 27, by the way, I remember thinking like, what the hell does my high school GPA from?
You know, 10 years earlier have to do with anything I'm going to do know. And I remember thinking, but it felt kind of Silicon valley sexy, cool to me at the time. So I was kind of really into it. But, you know, as you do this a lot, God, as a, as a people manager, you realize just how little those things predict on the job success period.
But especially because you're hiring them for a specific role in a specific culture at a specific time in the, in the company. So, um, Amazon, I think, had this, this early preference to kind of look at those as signals to kind of got you on the top of the funnel. I used to make the hiring decision. I mean, Amazon was very good about digging and probing.
Like we talked about even before bar raisers, there was just a real culture of show me. Don't just tell me, like, you know, demonstrate which, which lends to your, or it goes into your second part of your question there engineers have always had this, uh, you know, show me, don't tell me kind of orientation, go to the whiteboard and write code or, uh, find, find the bug in this.
Let's open your laptop up and have you do something for me. And they complimented it with, tell me about a time and stuff. Generally, they like to do more of the show, me kind of stuff. Uh, MBA, uh, hiring, which you were involved in, Dave, um, as well, there there's a, there's typically case studies and other things that non-engineers might use, uh, as well.
And what I found, I learned a lot of my own interviewing techniques by studying what engineers did even before Amazon. So if I was hiring. A recruiter, for example, or recruiting manager, I might give them a real time problem to work on. So a recruiter, I might say here's a job description. Here's a stack of resumes, stack rank these for me, and then pitch me the top two resumes as if I'm the hiring manager, or I might say, Hey, we need to hire 70 engineers in three months.
I want you to build out funnel. I want you to show the math and we, you to work through the plan, talk about sourcing, interviewing, closing strategy. And then after they do that, I start changing up variables. Now, what if that wasn't the United States? What if that was China? What if instead of 70, it was 700 and I kind of put them in a position where they have to demonstrate their capabilities, their IQ, and ETQ for me right now versus just tell me what you did at that company.
Some of that, tell me about a time stuff. It's helpful. They could still be kind of bullshitting their way through some of that. And I would have no way to know. Right. You know, you mentioned earlier where you're the manager or the underlaying, like, you know what I know if that could just be lying. So this is where you actually see much more of that, you know, present performance as a predictor of future performance.
Behavioral interviewing is looking at the past. What can I do now in the moment for you that gives you confidence. I'd be successful here. 
Dave Schappell: [00:36:34] Yeah, I think they're better. I think they're better at it now, even than I left in 2016. And I don't remember for the non-tech roles that we were asking, but I heard now that they're doing writing samples for pretty much every role, which makes sense in the era of six pagers and all this writing.
Um, so yeah, I, I, I remember, uh, you, and you referenced it too. It's sort of like at the end of your interview decision, it's like, will I admire this person? You know, or are they going to increase the quality of the team on some dimension? And if you, if you can't start answering yes to some of those questions, then you might just have.
Solid B player here, you know, and, and they're not really raising the bar on any, in any area and it means you have to look a little harder or longer. 
John Vlastelica: [00:37:20] Yeah, for sure. I, you know, Jeff, Jeff had four questions from that shareholder report. I know you'll put a link to that in the, in the, in the notes here. Uh, but one of the things I remember, I remember being so freaked out, David was my.
I dunno a month or two on the job. And I don't remember who it was exactly. I'm going to attribute it to, to, you know, Jeff Holden maybe, or Rick . So, you know, some leaders in the technology organization, but someone I remember said, look me in the eye, would you bet your badge on this person? I remember thinking like, holy crap, like I, you know, I'm a good interviewer, but I, you know, honestly I had an hour I'm like, would I bet my badge?
You know? And then I remember years later, someone said, well, you dilute the stock of the company to give this person a, you know, whatever their stock options are going to be, you know, is this the kind of person that's worth diluting the company? You know, you're, you're sharing the company for this. And I remember thinking, holy crap, like this is intense, you know, but that's the kind of bar raising mentality in the early days and had Amazon not focused so much on that had not been so evidence oriented and not done such a good job in the early days.
You wouldn't have scaled a high bar, you know, you can't start off with, you know, whatever BC player types, and then suddenly have a bunch of A-players. That's not how it works. You have to start off with top talent in the beginning to scale that up. 
Dave Schappell: [00:38:29] Yeah. And, uh, so I'm going to jump, I actually, I'm going to go a little out of order.
So, uh, yeah, we've, we've talked about bar raisers, making great hiring decisions. Uh, one or two episodes ago, we talked to Matt round and he built mats recruiting tool. Um, and I do want to talk about the tools a little bit, but first maybe talk about what you've been doing for the last 16 years since leaving Amazon.
Because like a lot of the people we talked to they're either retired or they're doing something totally different, but you've actually been living this in a, in a lot of ways. Right. Uh, since. 
John Vlastelica: [00:39:02] Yeah, I left Amazon. I had a, I had a brief stint as the head of recruiting media, uh, here in Seattle as well. But, uh, I spent most of the last, you know, 15, 16 years leading a consulting and training firm called recruiting toolbox.
And so a lot of the work that we do is to help companies raise the bar on kind of who they hire and how they hire. So we do hiring manager, training, recruiter, training. We do consulting projects to help define hiring standards. And we've worked with, you know, hundreds of companies in 30 countries, a lot of big tech companies hire us.
Um, and a lot of startup tech companies hire us, but we've worked in retail and banking and all the industries. So as you can imagine, um, but it's been really interesting, you know, when I first started there. I had this tendency to want to recreate Amazon, right. Because that's what I knew. And that's what I helped to build and had six years of that, you know, rolling around in my brain.
And it's funny day, because I will have people that reach out to me, I'll be on a zoom call with New York, or we have clients on Paulo right now. We're talking to someone in India a few weeks ago, and all these companies have some version of bar raisers or they want bar raisers, or, you know, they, they, they've heard, they've read about, um, Amazon's hiring practices and you can't just copy paste Amazon into another company.
Right. And we may talk about that later if you want, but, but it is really interesting to me. I don't run out there. I'm not out there trying to recreate Amazon at other companies. Right? There's some things about Amazon culture that some leaders specifically don't want to bring into their companies anyways.
Uh, but there is no question that Amazon is respected very well respected for having a high hiring bar at scale. And, uh, there's a lot of people that kind of look to that. Amazon rigor, even if the implementation of an idea is very different. So, you know, we helped Yahoo Yahoo when Marissa Meyer went from Google to Yahoo as the CEO, they were putting in Google hiring committees as an example.
And you can read more about that and like Laszlo Bock, who was the head of people at Google wrote a book. And it talks about that if you want to learn more about that, but hiring committees were an alternative to a bar raiser style program. And Facebook has kind of a combo thing with hiring committees and some special interviewers.
And lots of our clients have some version of something that's an inspection mechanism that make sure that the bar stays high. And doesn't kind of succumb to, you know, affinity bias hiring where managers just in their own image kind of stuff. Um, so a lot of our work is kind of around that, which is so the hiring bar hiring bar definition is probably.
A big foundation of most of your projects is that companies wrestle with that? Yeah, it was just on the phone last night with a potential client who said, we want to skip that and just have you do training. I'm like, well, you can't skip the defining that. What are you going to train people on? Well, we're too expensive and it's kind of a waste of time that you could do a one hour behavioral interviewing workshop and be done.
If you're trying to truly raise the bar, you have to define the bar first. You can't just so, so yeah, a lot of our work is, is, is around that. 
Dave Schappell: [00:41:41] But for people that are interested in this, I do, I mean, I would recommend you buy, you buy the working backwards book anyway, but that chapter is really good because what you were just saying.
You know, you can talk about, you know, you can't really hire for a position unless you write the job description well, and you know, you define what you're looking cause you basically John and well, Colin and bill broke down the seven or eight different parts of the hiring process, starting at writing the job description and reviewing resumes.
And, and you'll just see Amazon's approach to this. It's not a one thing. It's every step of the way where you're reinforcing. Do we know what we're looking for? And then are we looking or re evaluating candidates on that? We making decisions and the bar raisers making it. So it's, it's, it's basically, it's this cumulative thing that builds on every step of the way.
And you're, you're basically starting with clients at the very start of that and saying, if you're not willing to set the, define the hiring bar, then what are we going to do here? Because people aren't going to know how to, what they're looking for in roles and how it fits the company's goals and all that.
John Vlastelica: [00:42:42] Yeah. And you know, the, we call them hiring principles are different than leadership principles, but hiring principals, like I mentioned, hiring a brilliant ass or how do we think about pedigree or are we willing to make a trade-off between speed and quality? Like, are we actually willing to say I will sacrifice.
Right. Speed to make sure we hire the right person. And these are important questions. And it's not just an HR question. It's like when we do this hiring bar work, we're often working with CEOs or CTOs and heads of HR and diversity and all these other areas, but we're really working to try and align to see if there's an alignment at the senior level around some of these big, big questions, because good training is prescriptive.
You're not just teaching a generic skill, you're teaching the Amazon way or the whatever, fill in the blank, your startup way. And a lot of people. They assume we're all aligned on this. You know, I assume Dave, you and I would have similar, you know, whatever, we're both smart and capable people. Who've had a track record or whatever, but at the end of the day, your tolerance for, you know, the ass factor or your tolerance or your, the waiting you put in pedigree or something may be really different than mine.
And so I think organizations, especially startups, one of the best investments you can make is, you know, overinvest in defining your hiring standards early and then make sure that that's embedded into job descriptions. And you're not using things like pedigree and years of experience and prior job titles as hiring criteria.
Cause how do you interview someone on 10 years of experience you've done. It's about the quality of experience, not the quantity of experience. Right? Right. 
Dave Schappell: [00:44:04] And you, you mentioned something about. And last few minutes, you hinted at it, but you, you mentioned something in our pre discussion about who your customer was and you had an epiphany that, you know, is my customer, the company, the hiring manager, my boss, like, and can you tell that story about, I think it was a Rick.
I forget exactly how it went. 
John Vlastelica: [00:44:25] Yeah. So, you know, one of the things that I think is important as a, as a recruiter is, you know, there is a tendency, uh, I would say just as the HR profession to have to attract a lot of people pleasers. And so at a company like Amazon, where the customer is, everything it's talked about in every meeting all the time.
Right. I remember in my interview process with Rick dolls, L I could tell the interview started to go well, as soon as I started talking about the hiring manager, as, as my customer, I think he wanted to hear me talk about the customer in this, right? And for some companies, they talk about the candidate as the customer, whatever, one of the things as you're scaling though, as I was scaling my team, my talent acquisition recruiting team, um, I remember thinking to myself.
My God, you have high expectation, incredibly intense hiring managers. The most intense hiring managers I've worked with in my career probably has. I've never seen it's equal. Um, these people are not easy to please, like if I, if my metric for am I doing a good job? Is, is, are these people happy with me?
They're going to be unhappy because we had so many leaders in the technology organization that had unlimited appetite. I remember we hit a, a hiring target and I'm, I'm transposing them. I don't know if it's 2 68 or 2 86. Right. But I remember we hit our hiring target for the entire year in July. And I, and of course I knew I wasn't going to be out of a job.
I knew, you know, but I remember going in, I remember just like having that, you know, 18 seconds of joy before it's like pedal to the metal, like let's try and double that for the second half of the year. You know, you didn't get to, you didn't get to celebrate anything because it was just get big, fast, let's go, let's grow.
Let's do it. And so I think for me, you know, thinking about the company as, as my customer is also a way to kind of bring ownership into the recruiting function, because I started to realize, you know, this hiring manager may want this very specific thing or may really like their process to look different than the way, but that's not going to scale in the same way that having, you know, building a hundred call centers where customers can order like out of a catalog, wasn't going to scale at early Amazon, we needed to kind of build self service tools.
I need to start thinking about my team as how are we going to build something that works across this, this unlimited appetite organization. If I try and cater to every hiring manager. It's going to break. I can't do it. We have basic batch processing. We have to. 
Dave Schappell: [00:46:31] Yeah. The way I think about this is I forgot to frame it, but basically you help with defining the BARR training, interviewers, training, uh, managers, and then this would be sort of things you help companies with and training recruiters, right?
Because, uh, they might need that a company might say, Hey, we have a recruiting function, but we think it could use some help. And, uh, and do you normally get hired by CEOs or by like the heads of HR? 
John Vlastelica: [00:46:57] Uh, it's heads of HR and heads of talent acquisition. Some of our clients have thousands of recruiters like an Amazon, some have, you know, 50 or 80 or something.
And so usually that's, that's our buyer, but yeah, that's, that's a lot of building those consultative skills that are not just, you know, the hiring manager is my customer. My job is to please the hiring manager, but to say, Hey, you know, it, it's actually a bad idea to, um, you know, overpaid this one candidate this one time just to get this buddy of yours in here, it's going to really wreck the rest of the stuff we're trying to do, or, you know, having the tough conversations and even being able to push back.
Um, I definitely had to learn pushback. Rick dolls L sat me down once. Again, our CIO and I was looked to him as a mentor. I had seven bosses in six years and Rick was kind of a constant in there. And I remember we were eating, I was eating onion, rings, eating my feelings, feeling like quitting Amazon. Did you ever have that feeling, Dave, where you at days quit?
Uh, weeks, months, years at a time. Yeah, exactly. But I, but I remember Rick giving me some great advice. I don't remember the exact words he used, but it was kind of this conversation around the customer. Like basically, you know, you, you need to stop trying to just please all these individual people, because you're, you're, you're gonna, you're gonna feel like a failure.
And I was. I shared with you that New York times, quote, you know, the Amazon is the place where overachievers go to feel bad about themselves or whatever that I find that humorous, I think that's funny. Um, whatever the version of it was, but I found myself feeling that way. Like I'm used to being a high achiever and I feel like a total failure because I just cannot keep up.
It's just the weight of the world is Jeff Bezos is literally writing about hiring and every one of his annual reports, it's, you know, when I'm in meetings, people are saying our ability to grow as a company is dependent on finding more engineers or, you know, whatever I felt the weight of the world is a 27, 28, 29 year old, early leader in my career.
Um, and so Rick was able to kind of help coach me through some of that. Talk me off the ledge. 
Dave Schappell: [00:48:42] I wish I had had that conversation with Rick cause my wife would tell you that, that that was one of my problems is I solved it by just working harder and longer than exactly and gaining weight. I had plenty of that, that food, stress, stress.
John Vlastelica: [00:48:56] I say this now, you know, I, I reflect a lot on my Amazon experience. I learned so much. I know you did too, but. Um, I, after Amazon, I started having this saying, you know, try harder is not a strategy. You know, run faster is not a strategy. Work longer is not like that's not actually a scalable strategy. And so a lot of the work that, that I do with companies now, but a lot of what I reflect on, what I learned was just, you have to have mechanisms like processes, your friend, or can be your friend and you need to have, you need to know what to say no to, so you can focus on the right things and all these things that I reflect back.
I learned that I didn't know what I was learning at the time. Um, just had a huge impact on how I think. How many companies now.
Dave Schappell: [00:49:33] I just interviewed Neil Roseman. So that, that episode will be a couple of weeks before yours. And he, he had the quote very similar of you can't, you can't rely on heroic efforts.
Cause if you, you know, as an organization, as your, as your default, because if you do, you eventually ended up with just a lot of dead warriors, you know, or dead heroes. 
John Vlastelica: [00:49:53] And I gotta say in the beginning it was a hero culture. I mean, it was the save the day. I mean, I, you know, my, my first child was born at Amazon, uh, you know, first year I was there and, uh, it, it was intense.
I mean, it was, it was intense, intense, and there was a hero culture and you worked harder and you believed it wasn't because. You know, I had to hit my vesting date. It was, I really believed in what we were doing, big cheerleader for the company. And it was a company that really lived its values. And that was unusual to me, you know, I'm, I'm, I'm an extremely cynical person having come out of HR.
I look at a lot of shit that comes out of HR and I'm like, I'm like the engineer who just rolls their eyes. You know, that was me. I was always the odd ball who was like, this is going to come off. So are you really going to hit, send on that email? Like, you know, and so going to Amazon where I thought, wow, we're actually living these values.
Like this is a real, this is the real deal. It was, it was unusual. And I don't see that as often as I'd like, when we work with other companies. Yeah. 
Dave Schappell: [00:50:47] Well, I'll challenge that this is a good segue to, uh, the technology tools that the recruiting function had. So yeah, because we really had none or maybe I'm being cynical there.
Uh let's I I'd like to. So Matt round in the episode, 10, I think talked about the recruiting tool and he literally built it because the tools for managing the recruiting process were so horrendous. Um, I think. Literally, we were emailing resumes around and trying to track ratings and PeopleSoft was either slow or not working.
So, you know, we weren't living our values in terms of obsessing over the recruiting function as a customer, given how important it supposedly was. So let's talk about that a little bit and, and then really let's get to the positive part of it. Uh, just cause I tend to live there, but, uh, yeah. You know, how, how are companies that you work with now dealing?
I'm assuming the tools are light years better than they were back then, but it was pretty sad what we had to deal with for a long time, not like six months, I'm talking about six years. I still felt like we were wrestling with the tools for managing all these processes. Maybe you'll tell me. 
John Vlastelica: [00:51:54] And I'm going to say, this would be funny. Spring 98, I started, we had a thermal fax machine and you need to Google that to even know what that is. A thermal fax machine with rolled up paper, where resumes came in on the ground. We weren't even having people email resumes to us, which is just bizarre, even for 98. Right. And we had in-person resume reviews.
And so, you know, we would print out, uh, you know, resumes, make six copies and have, you know, Kim rock Miller, Neil Roseman, all the people you're talking about and talking with would sit around and review that progressed to getting an actual, you know, system in place. But it was clunky, uh, not many, not many products out there were amazing.
And so eventually. Round, you know, I think he did do it in a weekend. I listened to the interview, but you know, his first iteration was to solve a specific problem. And, uh, and it was honestly, it was really built from a hiring manager's perspective. It was not the best tool for my recruiting team to use.
Um, it didn't really have some of the components that are in, uh, even, uh, a great tool at the time would have, but it was absolutely amazing. Absolutely amazing what it did. It did really well and get some development support documentation, and, but absolutely amazing. And so the, the mat round tool or Matt's recruiting tool MRT, uh, for example, had, had really cool mechanisms to allow us to share.
Resumes in these pipelines, people could tag them. Uh, we could get feedback on resumes. And most importantly, it was a great way to collect interview feedback and had some little mechanisms in it. Like I couldn't read your interview feedback, Dave, until I submitted my own to try and remove that kind of group bias that might be in there.
And you gave feedback on feedback. So I talked about this earlier, the bar raiser in the hiring meeting would call you out. Dave, if you did a crappy job of saying I'm a thumbs up and you couldn't back it up that the tool, um, gave people the opportunity to rate feedback. So I could read your feedback, Dave, and if you wrote three sentences and you just kind of use broad generalizations without evidence, I would give you a thumbs down and you developed a rating as an interviewer.
That is all part of this culture of how do we build better interviewers in the company and have feedback mechanisms and inspection mechanisms. So in that respect, It was amazing. It did some other things over time, too, but that alone, I still don't see systems out there in 2021 that, that have some of the capabilities that Matt built, you know, 20 years ago, which is just crazy.
Dave Schappell: [00:54:06] It is. I mean, like it's, it's, it's also pretty crazy that in the midst of all the things Matt and Josh were talking about, their teams were building, you know, Matt saw this need. Cause he was probably doing four interviews a day and you know, 12 a week. And, and he, and he said this, so he just invented email.
He invented a solution that solved his problem and the problems of all the other interviewers, not to your point. It wasn't built. HR. It was built for the people doing all the interviews, but, uh, and he just built it. And I agree with you. I do think he built V1 in the weekend as well. That story was so, but, uh, yeah, he, he basically just said it.
John Vlastelica: [00:54:41] He's an amazing, yeah, he's an amazing human and absolutely an owner and clearly something outside of his quote day job. Right. But, but that was the thing about Amazon is recruiting. Wasn't viewed as something outside your day job. It was part of your job. And, you know, we dove into bar raiser pretty early day, but I wanted to mention, you know, for me, bar raisers was an ingredient in the secret sauce.
The secret sauce was the ownership culture. The secret sauce was that I didn't have a 20 person tech recruiting team. A 500 person recruiting team because everyone in the company, even at scale, even when we went beyond this kind of startup kind of feel to something bigger, everyone had expectations on them that you're, you're out there to help us get great talent of the company.
Um, and that is something that I see in a lot of startups, but you lose that as you get bigger. And, and the notion that today, you know, Amazon, as you know, many thousands of people just in the recruiting function, like you said, over a million employees, that's incredible. But the fact that Amazon is still scaled that were part of the core values leadership principles.
Part of your performance review is around yours. You know, capabilities and your results are alive in recruiting as a people manager, not as a recruiter, but as a people manager, that's ridiculous. Amazing to me. I lazing tell. 
Dave Schappell: [00:55:49] Actually a good story that splits those last two topics. So tell the story about how you pulled the data from.
The recruiting tool, shared it with an employee's boss to tell them how good of a job this employee did. 
John Vlastelica: [00:56:02] And then one of the things I was trying to reinforce, I was, I was constantly thinking about how can we create more? How can I reinforce this amazing call? I didn't build this culture. I inherited an ownership culture, but I was really good at iterating taking things I had and saying, you know, I had very little budget, um, very small team relative to the need, but how do we leverage all this great stuff?
And so I thought, you know what I'm going to do. Everyone hates performance reviews managers at every company, not just Amazon, right? We hate performance reviews. We're always looking for stuff we can say about someone. Even if we liked them, we struggled to write all this stuff. And so, you know what I should do.
I should run a report and see how many employees made employee. That were successfully hired. How many people went and did college campus recruiting visits and got good feedback. You know, who was a bar raiser? This, this quarter, this year I went through and pulled all these stats that are not around recruiter stats, but their recruiting sprouts statistics or metrics that are aligned to individual people, managers, and employees.
And I thought, you know what I should do with this? I should send this out a few weeks before performance reviews are due to the manager of the person that I'm kind of praising and CC the person, but send it to their manager and say, Hey, just wanted to let you know, Dave did an amazing job. He made three employee referrals that were hired, Charlie Bell's, Oregon, Neil's Oregon, you know, whatever.
And then here's all the other things and no surprises. Many of the people that I sent that to responded back. Thank you. And then put that information into the person's performance review, which is one of the ways you sneak, hiring into performance reviews and create that culture of ownership and recruiting ownership.
There was this guy, Jeremy, who was killing it. He was the guy. Oh, my God, we had a bar raiser call in sick today. We need a backup and the candidates already here, and this guy would jump out or whatever he was doing. And he would make time for us and we could plug them in and we took advantage of him probably more than we should have, but he was absolutely fantastic.
So he comes into my office. He's one of the people I sent an email to, you know, describing all the great things Jeremy had done. He came into my office and in a very sarcastic tone said, thanks a lot, John, for sending that note to my boss. And I'm like, well, you're welcome. You're amazing. What, what, why are you being sarcastic?
And he said, no, he put that on my performance review. I'm like, awesome. You're amazing. He goes, no, I don't think you understand. He put what I did last year, my performance review. And now I have goals in my performance of you that are your goals. Dude, you put your goals in my performance review. Like what the F you know, what are you doing?
I don't want to have, I don't have to make six referrals next year. Like I was doing that. It's just the general owner and what the hell? You know? And so we, we kind of laughed. He was giving me shit, of course, you know, but, but he was absolutely to me, I just, you know, you can imagine the grin I had because.
What if we lived in a world where every hiring manager, every employee was rewarded for this in their performance review and even had some goals, wouldn't that be incredible beyond just living the values, but actual numbers. And so that to me was a great indicator of the kind of thing had I stayed at Amazon.
I would have really wanted to scale up and just kind of embed numbers and everyone's reviews. 
Dave Schappell: [00:58:45] So yeah, I would've started inserting product development goals in your, in your reviews. So giving you some work, um, we're almost at an hour with three minutes away, so, and I still have a few more questions, so we're going to go over.
So you work with a ton of companies and, you know, you mentioned Yahoo, you were at Expedia. And so what do most companies have the most difficulty with? And I'm not talking about applying Amazon, I'm just talking about, you know, you mentioned a bar definition, or I'm sorry, basically defining these things, training recruiters, training people.
What are the things that the places most commonly Russell. I, you know. 
John Vlastelica: [00:59:21] There's a bunch of things. Um, one of the things that I think is we talked about already is a lot of, uh, companies assume that because they have posters up on the wall, what their values are, their competency model is for, for, for performance reviews, that those are being used and used appropriately in the interviewing.
And so, um, I think one of the big opportunity areas for companies, if I were giving advice, certainly to, to, uh, you know, an entrepreneur or a CEO, it would be make sure you have really defined what good looks like, and you've got your team aligned on it. And you're actually have, uh, an inspection mechanism to make sure that it's being used appropriately.
And one of the ways you test for that at Amazon, we use bar raisers. You wouldn't just be able to say, I thought they were good for the company you had to back it up. Right. So have some kind of inspection mechanism. Um, that's one thing, there's a lot of organizations that have not built a recruiting organization and recruiting capabilities to achieve their goals.
They're just, you know, HR has historically been incredibly under-resourced and I'm working with a company. 
Dave Schappell: [01:00:18] I always say an Amazon was too, just to share. I mean, right. You guys were so lean given what you were being tasked to do. 
John Vlastelica: [01:00:24] And I think about it. Yeah. I mean, if I could go back and change something, we should have had more, more recruiters and more resources for sure.
But one of the things that I tell people today is, you know, you want to make sure that you build the size of recruiting organization for the company size. You plan to be not the company size you are today. Yeah. We have a client right now that this is really interesting to me, but you know that they have 800 employees and they have a 50 person recruiting team.
We have another client. We have two other clients that are about 2000 employees, and these are companies you've heard of Dave. I won't say their names, but 2000 employees, each, they have a hundred person recruiting department. So one in 21 and 18 employees are on the recruiting team. And that is crazy because I had 20 people, you know, at one point to support, I don't even remember what the numbers were, the size of the company, but I, you know, many, many thousands of employees and crazy hiring goals.
And so what I'm seeing now in 2021 is people are resourcing their Tonya acquisition functions very differently than they did before. We used to have hiring targets for recruiters, have, you know, five, six hires a month. Now people are one or two hires a month. Um, and so there's just a different, different kind of recruiting environment going on, but those are two things that are really important.
And of course, you know, making sure that, you know, you are getting feedback, you have some kind of feedback mechanism. Uh, so, you know, candidate feedback, you know, making sure you're, you're, you're, you're learning how appealing a lot of companies. They think that, you know, everyone's dying to work there because we've all drank the Kool-Aid.
We all believe in the brand. We all see the vision, but candidates, especially top technical talent, top enterprise sales, talent, top product talent. They have so many choices right now. And so many companies I think are a little blind to their, their actual appeal. Um, and you need to have a mechanism in place to understand how appealing you are.
And a lot of times that doesn't mean you don't make your career site fancier. You sometimes you have to fix the inside before you fix the outside. And I've talked to some companies where I said, I appreciate you want to only get talent out of Google, Facebook, Facebook, Netflix, and Amazon. But I don't know that you have engineering projects that are challenging enough.
That's going to attract that talent. Somebody you can even afford to extract them away from their RSU packages and, you know, good luck getting them. And what are they going to do? You know, I don't know if you have the story. Well, instead of humble, is that what you're looking forward to. 
Dave Schappell: [01:02:44] It is I was going to say the fourth one, which I surprised you didn't get to, but I'll, you'll be like, how did I not say it?
It really is making every employee and owner of recruiting in this bar. Okay. It it's, and that's why, when you were just giving me those numbers, I'm like, that seems like a lot of recruiters for a company that size. I was, I wasn't sure if you were going to say that's not even close to enough. Um, but yeah.
Everybody has to treat recruiting as it's easy to say, it's the most important thing, but it seems to me like those companies have learned what we're talking about today. Like the impact of hiring a Marian Mohit or , or, uh, you know, Neil Roseman or Kim rock Miller. That impact is, I mean, it, it, the ripple effect is, is infinite.
Like I talked to Eric Benson yesterday. His will be the episode before you. And he and Bob had nasal, I think Bob also helped with the development of bar raisers, or it was one of the first, one of the first 20. Yeah. He talked about the project they did. And probably a lot of engineers worked on the port.
I mean, that project has probably had a multi-billion dollar impact on Amazon. And it's because you had these people that were in the midst of the chaos of everything were, they were also top recruiters. They were bar raisers, they're doing these projects and it's because they treated everything like they were owners.
And, um, and so I, I kind of think like a lot of what you did with these programs, whether it was intentional or not, or it was, uh, an outflow of, it was just getting everybody acting and mentoring and training and becoming owners of the culture and, and the, the, you know, your facts. 
John Vlastelica: [01:04:17] Yeah. As I said, that's the secret sauce, man. It's the bar raisers were one ingredient. But the, the ownership culture, when I, when I talk, you know, I presented conferences, uh, you know, about recruiting best practices, kind of stuff to my peers in our industry. And when I talk about, you know, the, the single biggest thing you can do as a talent acquisition, right.
It's kind of help scale a high hiring bar. I don't talk about bar raisers. I talked about creating a culture of recruiting ownership, where everyone feels connected to it. That's how, especially now in the age of social media, where you have employees very easily being able to share their experiences online and stuff like that, it's just, it's ridiculous not to leverage your employee base and people want authenticity and they want social proof that this is actually a good place to work.
So they're actually, depending on, on employees, I remember Bob bad named particular, you know, he, he was one of the guys that slept under his desk, um, on more than one occasion, I'm sure. But I remember meeting him in my first few weeks and, uh, you know, asking about the futon or whatever it was and just kind of learning around, you know, just how, how intense this was as a, as a place to work and still realizing that he like everyone else was so invested in getting great talent to the company.
I mean, they had a personal vested interest. No, these were all, these were top people that wanted to work with other top people. So even if I had some kick-ass recruiting team that was five times the size. Even if I could have been the most customer service-y, you know, recruiting team and please every man there still needed to be that, that ownership culture to get the kind of success that the Amazon had it.
I think that is the bigger difference maker. Certainly more than anything I did. I can't really point to many things that I did that anyone else probably wouldn't have done it they've been in a similar situation, but I was really good at seeing things that I could kind of build upon as opposed to coming up with stuff from scratch.
A lot of it was like, oh my gosh, if we can reinforce this ownership culture using MRT little icons, or if we can put, get this put under performance reviews, or if we, you know, if there's little things like that that are just kind of reinforcing this thing that already existed magic happens, you know, it's just, it's.
Dave Schappell: [01:06:14] Well, thank you so much for being my guest on the podcast. Um, I am I'm positive. People are gonna love this one, because like I said, this is where almost all the feedback I get from startups is asking that question about how in the hell did they hire all these, these unbelievable people, because we weren't paying that much.
And the stock options at the time weren't worth anything, you know? 
John Vlastelica: [01:06:34] Oh, my gosh. Yeah. The number of people that hung up on me when I was operating the mace, you know, going backwards by $40,000 in their base pay or taking, holds back and level, you know, titles and stuff. Yeah. It's really interesting. But yeah, I agree with you, Dave, the people you've been interviewing have just been phenomenal and I'm so excited.
I love the podcast. Uh, and I also look back on that. I think like you do. I Amazon made me smarter. Like there's no question. I, I got the most dense experience of my life there. My first two years there, I got about 10 years of experience crammed into it. Um, I was a total under hire. I still don't know why the company hired me.
I was not nearly, I was the dumbest guy in every room for so long. I probably still would be. Uh, but 
Dave Schappell: [01:07:15] Hey, I was just happy when you got, I was just stopping when you got there. So there was somebody dumber than me. You know. 
John Vlastelica: [01:07:22] There was a lot of that, the feeling of like, am I ever going to work at a place that's gonna make me the smart, like, like poking holes in your work and having people that are not trying to call you out to be an ass, but they're really trying to make you better because they're not.
They need you to be better at your craft for them to get what they need. And that was just, it was just incredible. So I really appreciate the opportunity to share this stuff. 
Dave Schappell: [01:07:43] Well, thank you very much. Um, for the audience, thank you for listening to the event like an owner podcast. If you'd like more details about what we discussed today, and I will be posting some of these original bar raiser documents and things, uh, on the website under, um, John's post.
Uh, so if you want to contact me with edits or help us with the details, please visit invent like an owner.com to sign up for the weekly newsletter. And that's it for today. Remember? No sniffling. .