ILAO 10
Josh Petersen: [00:00:00] We put cart racks out there. Um, and we had the test results. There was still, I would say, highest paid people in the room. Still thought you can't mess with your shopping cart. You can't do this. And remember that one going quite high up for final approval, but the metrics were all there, but it's so early that people can't mess with shopping.
Dave Schappell: [00:00:31] this episode is sponsored by Ideoclick led by ex Amazonians, Justin ley, Trump furphy, Andrea LA and Mike Burrington idea. Click is the leading e-commerce optimization platform for brands to win Amazon. And e-commerce. Over 350 brands on Amazon. Rely on ideal click for e-commerce strategy, advertising execution and maximizing growth and profitability.
To learn more, visit ideal click.com that's ID E O click.com. Today. I'm thrilled to be talking with Josh Peterson who joined Amazon in 1996 and Matt round who joined in 1999. Uh, they were the first two leaders of the integrated multidisciplinary personalization team that evolved amazon.com from a site that was largely the same for everyone.
Think of how a physical store is laid out, where we all get the same experience to one that fulfilled Jeff Bezos's vision of a shopping experience. That's customized for the interests of every unique customer. Today. You'll not only learn about the early innovations that created Amazon's personalized shopping experiences.
But maybe just as important Josh and Matt will talk about the unique organization structure that the personalization team implemented and how that team format accelerated their impact and transformed how Amazon deploys teams to this day. Welcome Josh and Matt. 
Matt Round: [00:01:50] Hey, great to be with you today. 
Josh Petersen: [00:01:53] Thanks Dave.
Dave Schappell: [00:01:54] So, uh, just to frame this, just because again, my mom's listening, she's one of my most loyal listeners. Uh, can you just talk about what personalization means and, and especially in the context of 19 96 97, uh, why it was a big deal? 
Josh Petersen: [00:02:11] For me I always start with Jeff's vision because I think this is an area where Jeff was really particularly influential and really gave us the vision of a store for every customer.
Like you said, and, uh, I guess to give you some sense of how scale was so different back then than today. I remember Jeff saying, if we have 10 million customers, we have to have 10 million different stores. And at the time, that sounded so crazy, but to have them personalized, but also to have 10 million customers sounded nuts.
But, um, that was the, that was the vision you could say. We worked backwards from, and then you, you asked about 96, 97, really early days of, um, personalization at Amazon. I think of really two features that were key. And one is similarities that I'm sure we're going to talk a lot about which, uh, in a way is the same for every customer, but it feels individualized because you're on a page of your own interests.
And from there you can see other people's buying habits, which again, those always use, uh, purchase habits. But because we started out just as a bookstore and later music store, the data felt personalized because they've represented similar tastes. You know, I don't, I don't know if you're on toothpaste today, if it feels particularly personal to see some other items may be useful.
And then the second area at that time was effort around a system called bookmatched. Um, that was really some licensed technology doing collaborative filtering, um, licensed from a company called group lens. And it was a very different model in that it had you interact with a list of items presented to you and asked you how much you liked them, and then put you into affinity groups with other people who liked to items similar to how you like, right.
Those were the first two early approaches was before, before the P 13 N team was formed. 
Dave Schappell: [00:04:04] Got it. And was, was a bookmatch or was that you mentioned group lens. Was that the net perceptions that people have said about same, same company or same, same technology. 
Josh Petersen: [00:04:15] Instead of university of Minnesota? I think, um, it got commercialized and I know JJ was leading that effort. I can't remember everybody who worked on the project, but JJ and Eric Benson and. Greg Linden and Andy Hardebeck what sort of a early tech team working on group blends and similarities. 
Dave Schappell: [00:04:36] Yeah. People keep asking me to describe who people are and what they did. So I'll just quickly try to do that quick. JJ Jacoby was, I guess, the first head of personalization, uh, or what were her other titles, community multimedia.
Um, Greg, uh, Eric Benson was one of the first, uh, called senior engineers. You guys, uh, correct me, Greg Linden, the same, um, superstar engineer did a lot of work on personalization. And then what was Andy Harvick's, uh, function or title you. 
Josh Petersen: [00:05:07] Started as an SDE in that team and then stay all the way through personalization until he worked in Jeff's office, I think ultimately started up satellite team.
Dave Schappell: [00:05:17] And so with those first features or, or, and I'll just use bookmatch as an example, cause I, uh, I believe that ended up not working longterm where they. You know, where they judged primarily on incremental sales or was it, you know, on a user experience, like sort of a softer user experience thing, or how, how were all these sort of personalization features evaluated?
Josh Petersen: [00:05:41] I think, well, I'm gonna answer you a little bit roundabout, cause I, I think one thing to think about is that this was a team of all STEs creating a customer experience around recommending books and a part of what we did differently in P 13 and has made it more multi-disciplinary and a little more, I would call productized where we presented a customer experience that was sort of complete and worked through the site and tied to metrics.
So I say that a bit because I don't know the answer to your question about how it was evaluated. 
Dave Schappell: [00:06:14] Well, just to clarify, actually, I mean, just because, um, most people, when you're saying is the first people that we're building, the first features that you were talking about were primarily all engineers. And whereas when we haven't gotten to describe the single-threaded team concept for your, the first personalization team.
So, so those first features that you're talking about things like similarities, which are Benson launched and bookmatch matcher, which JJ, and, uh, Greg, maybe Greg, I'm not sure. Or, you know, uh, in some recommendations, which I know Greg launched, those were all more technically. Or how would you just, you know, distinguish them where they early personalization efforts that were, you know?
Josh Petersen: [00:06:58] Yeah. I don't know if, I think maybe I would say something about what Greg really saw in a shortcoming of collaborative filtering. I think this is how I remember it anyway, is that you had to have a predetermined list of. Presented to you that you could then rate, right? So you've got these things about, you know, what do you think of Shakespeare?
And everybody thinks eight Shakespeare, even though no one knows anything about it, you know, or what did you think of, uh, you know, wizard of Oz, who's this re reproduce list that wasn't really relevant. And meanwhile, there was the similarities database. And I think Greg saw that we could use the similarities database to give better recommendations than if we were using the candidates, um, from, from the collaborative filtering.
So this is right. So I might, I mean, Matt could probably do a better job than I could describing, you know, that the technical piece of what item, the item similarities look like versus collaborative filtering. We shall see if we can. 
Matt Round: [00:07:59] Um, so similarities we keep talking about is this idea that people who bought X also bought Y um, but it's a little bit more sensitive than that because, um, everyone bought Harry Potter.
So, um, you can't simply, you have to look at kind of how surprising it is that people who bought X also bought Y and then when you've got that kind of item to item linkage, I think that the crucial insight of instant Rex was, um, when people buy something, that's a completely different kind of. Um, commendation or connection to a product then rating and five stars.
And so working off their purchases and then, well, people who bought the things you actually purchase, not just things you say you like, but things you'll open your wallet for the things they bought as well are likely to be interesting to you as well. And so I think that was the, the, the, the big kind of leapfrog, the instant Rex did using that, um, purchase based indication of intent rather than just a kind of abstract rating.
Collaborative filtering is a little bit more like, um, trying to put people together rather than trying to put items together. I think that's what net perceptions or, um, group lens were doing. I think they were trying to gather you with people like you, uh, which ultimately we did do something along the lines.
I became soulmates, I think. Um, and, uh, never quite the breakthrough that it could have been. 
Dave Schappell: [00:09:19] Yeah, it is, it is interesting. Like you talk about the Shakespeare problem, you know, it's when you join a new service and they say, list the artists you like, and you end up just clicking everything. You're like, oh, I like this.
And I like that. And I'm like that. But if they look at the things I actually purchased, it's a little bit more embarrassing. Like, oh, I actually only bought country music. Uh, what's going on here. Um, 
Josh Petersen: [00:09:40] Dave, I need one other piece of, you know, what was different than was, uh, you know, really we had the team, we had the whole company in silos, you know, so we had a design team and editorial team.
Uh, we had a Ste you know, a software team and it was really hard to ever get anything all the way out to the site without begging and borrowing people from the silos. And the one time it was always different was when we did a product launch. Um, you know, so there was sort of a big enough effort, like launching music or, or auctions, you know, you, you had permission to go borrow from everyone to put together your team, right?
And, um, so when you ask, like how was it different before and after? I do think that was part of it is that we started running with the idea of personalization to try to build out just vision of a store for every customer, um, rather than sort of waiting to be told what was the next company effort. 
Dave Schappell: [00:10:33] And was it, was it the launch of auctions that was sort of the, the thing that was, that was that the pivot point between having a dedicated team or a fully threaded team versus not? Or how did, how did we go from this being more driven by, you know, engineers like Greg Linden and JJ, uh, Eric, you know, building functionality and features to saying how, you know, we're going to have web dev design product, all, you know, resources, all in one team, um, to focus entirely on personalization as a, as a core function for Amazon.
Josh Petersen: [00:11:11] Yeah, I, I think auctions had a part, but largely accidental. So one part was that it had us all working together across discipline to get this big push, to get the, the auction site launched. And the second one is I think it was really harsh when I say that first one. And it's probably the first time I started working directly with Greg and I were working on how to log in across these two experiences.
And it just put us into more direct conversation around personalization topics as well, not because of auctions, but just because of proximity. Right. Um, but the, I also think it was such a death March, that there was an understanding that when this was done, people would have, uh, some, some ability to choose what they worked on next.
Dave Schappell: [00:11:55] Got it. Um, so let's, let's talk about, we've been sort of hinting at it cause we all know about this. Can you describe the big change? Uh, when the team got, you know, we see after they auctions death March and we were totally successful, wiped out E-bay and moved on to dominate online auctions. Uh, well maybe, maybe that didn't quite all happen, but like talk about the team, you know, how, what the team looked like.
W w when you got the approval, um, to put together a team that's going to be focused solely on personalization. 
Josh Petersen: [00:12:30] Well, I mean, it is quite a, it was, I don't know that there was ever any approval or anything I'd say, um, you know, JJ was very generous to let people kind of work in her space. Cause she was, she had the software charter and the, and some of the most talented engineers, Ryan Snodgrass and Andy heartbeat, great London working for her.
Um, And sitting around Amazon was also this incredible, uh, purchase of, of a planet. All, you know, one of the initial, uh, social networking companies that was ever created. So like, if you think auctions was a missed opportunity, there is a big one there. And so Warren Adams and Brian, Brian Robertson, and others from that team, um, were, were also sitting adjacent looking to work in community.
So in some ways it was a stew of those four developers, um, Marianne Mohit, really giving us kind of space to see what was going to happen. So that that's the, and Greg, you know, Greg and I worked, um, and Todd Gaiman is the other first three people, um, working on that. It's, Todd's a web dev. And, uh, I think it's key to say that because again, I think it was really hard to.
A developer could have a great idea, but what about a web dev at that point? He couldn't get it out onto the website easily get it measured or tested. 
Dave Schappell: [00:13:55] Yeah. And so what was, uh, w when the team got set up, uh, or got put in the room on, in the Columbia building, uh, and you were all like, what was the tree?
Yeah, I think so. It was right next to Andy and Jason. What was the, what were some of the first wins? And I actually wanted to ask about some of the inputs you mentioned in the notes, uh, but ref tags and, you know, sort of how ref tags and web labs worked. Cause it's going to be part and parcel for how you ended up, um, continue, you know, documenting wins and continuing to impact or develop Jeff's vision of a personalized site.
So like, what were some of the first wins that came out of getting the team set up as a single threaded, uh, group. Matt, I think you were there for most of these. 
Matt Round: [00:14:46] Well, I mean, I only joined in 1999, so, so I think I, I never went to the Columbia building only PAC med. So, uh, um, and uh, if I was thinking about kind of the earliest features we launched, um, um, I remember working on porting customer reviews onto the German and the UK site and things like that to begin with.
Um, yeah. So the instant break system, um, what did we do next after that? I'm really struggling. This is 20 years ago. So I do apologize. 
Josh Petersen: [00:15:21] That's great. Again in reverse and talk about the metrics these first, a little bit. It's like we formed, I don't remember anybody telling us we could start working on this stuff the way we did.
And part of the reason we were able to was we were just completely focused on, on metrics because the degree we were inventing or using. Forms that other teams have created to measure our own success. Right. And the two key yeah. Are rest tags kind of amazing that, you know, this is long time ago, people weren't tracking, um, which features were generating which results, and that really became our bread and butter.
And maybe did it give you a sense of how fundamental it was? Is that a sales report at Amazon in 1998, might've looked like books sold this much music. So this much, um, you know, videos this much, but by, you know, six months later it was like search sold, 85% similarity sold 6%, right. You know, on and on. And we sort of thought to get from 6% to around 14% when I left in 2000, the beginning of 2002.
So, and in a way, what was so different too, is that we, we pivoted it to looking at what was actually generating sales and looked at more ways to do that. Can I ask about beauty product. 
Dave Schappell: [00:16:38] Before we get to that. Can you like, do you know who launched ref tags? Was it launched because people realized they didn't know which things were leading to, you know, which inputs were, were leading to which outputs or which features were leading to which, which sales.
Josh Petersen: [00:16:52] It's a great question for, for Greg London. I mean, I know what I would say Greg absolutely did was figure out ways to, um, get through all those logs and build, um, ways to see what, what the results were. It would, it's a good one for Greg. Okay. No, and same with web labs. I think we've mostly understand that at the time.
Dave Schappell: [00:17:12] Can you, can you just, just, can you describe what, what labs are again? Sorry, just to keep just for people who don't know, um, 
Josh Petersen: [00:17:20] I'll kick it to Matt again, cause they'll do a better job giving you a technical description. 
Matt Round: [00:17:24] So the idea with the Weblab is that you can do random assignment of arriving customers to control the old version of the website or treatment, the new version of the website.
And then you can let them live through that version of the website and look at whether the control group or the treatment group ended up spending more or less than each other and what that lets you do. cause other ways of testing, you could say have the old version last week and the new version this week and say this week we sold more, but it might be Christmas.
So it might be nothing to do with your features that you saw before. So it's really important to test both at the same time, with the same kind of customers, randomly putting people on one side or the other and then the Weblab lets you answer all sorts of questions. Like did sessions last longer, did people add more product to cart?
was cart ads turn into. Ordered items where they're more refunds, even on this side, did people buy more toys over here or over there? And so, Weblabs give you the ability to measure with very great confidence, precisely the impact of the change you have made and they let you filter out everything else.
You know, the time of day, whether it's raining, is it summer who cares because for all intents and purposes, your control group and your treatment group are the same customers. You know, they're completely indistinguishable apart from having had this different site experience. 
Dave Schappell: [00:18:40] And, and this is really common now, do you know, did Amazon patent that or was that something that somebody else had done before Amazon?
Like did Eric, sorry, there's two questions here. Do you know, did Eric Benson build web labs or do you know who did? And then the second question is, do you know if we have a patent on that or is that something I have to track down to discover? 
Josh Petersen: [00:19:01] Yeah, I don't know either. 
Dave Schappell: [00:19:02] Okay. Well, that's fine. 
Josh Petersen: [00:19:04] It took a while to get. I mean, one, one thing to tie it back to personalization, I guess I would say is that this web lab, I think everyone thought, well, we'll use it to test two different designs of the page or, or I don't know if it was Kylar, but somebody eventually thought, well, what if we should have different pricing? And it was a bit controversial.
And, uh, but, uh, what we turned it into was a way to deploy features to the site. Um, cause it was just agonizing to get people to agree, to let us take space on the site to express on homepages or detailed pages. But if we could show to the show, the successful web lab, the poll argument went away for the most part.
Yeah, it might get elevated, but if you showed that to Jeff or David Risher at the time, what if they didn't have that much to argue? 
Dave Schappell: [00:19:51] And for the listener, this is where this starts to all tie in. So w you know, without ref, without ref tags, so, and probably ref tags combining into web labs, it's hard to, to measure all this.
So. Let's talk about some of those first tests. And I, and again, my sitting outside of your office view was you guys were saying, we're going to put this personalized widget on the homepage and in, you know, in the shopping card. And so let's just talk about some of those, cause I know they were contentious.
Um, but I also know why this stuff like Jeff's head, must've been exploding with joy because he could, you know, he could just put these things in front and rather than argue about it and have opinions, but just let's see what works best. And so I'd love to hear about some of those. 
Josh Petersen: [00:20:39] Well, some of the, um, I think two key pieces, again, that Greg is very directly involved with around just similarities and instant reps, you know, a way of basically thinking, I think it accurately, Matt is you can take a seat Jason and say, show me essentially the similarities that are around this ACE.
And maybe later on filtering out things you've already purchased or gifts or others. And so that gives us instant wrecks. So, like you're saying there's, there was definitely a goal to put that onto the homepage to get right front and center to start building a store for every customer. But some of the biggest wins.
I mean, that was, that was sort of a technical achievement. We can talk more about, Matt could tell us about what was required to get it onto the homepage, but some of the biggest wins were things like a cart Rex, which just takes the things that are already in your cart  and builds like the most personalized, um, point of purchase impulse buys.
You know, that was a, that was a huge win. That's kind of amazing knowing that, that wasn't the first thing we thought of, right? 
Dave Schappell: [00:21:40] Yeah. Like some nowadays it's like, you have a printer in your shopping cart and they're like, Hey, do you want to add in paper or toner cartridges or whatever else it's, you know, but back then it was, yeah. This book, you might want to add this other book. Um, and it showed, I guess, pretty immediate big lifts in sales. 
Josh Petersen: [00:21:56] I think that was, I think that was probably the biggest, you know, quick wins, right? 
Matt Round: [00:22:01] The other one. I really remember. Do you remember by X get Y um, where we took the, on any items, detail page, which at the very top similarity, the thing most people bought with it, and we created a new buyer button that said, you can buy this item and it's top similarity together now for the same price, just click this button.
And that was a phenomenal win. I think it was 2.4% increase in site-wide GMs or something terrifying. 
Dave Schappell: [00:22:27] Well, the funny thing about that too, it's do about how early it was. 
Josh Petersen: [00:22:30] We've put the cart racks out there. Um, and we had the test results. There was still, I would say, you know, highest people, highest paid people in the room.
Still thought you can't mess with the shopping cart. You can't do this. And, uh, I remember that one going quite high up for final approval, but you know, the metrics were all there, but it, it was so early that people thought you can't mess with the shopping cart. You can't. Today, it's hard to find your way out of the shopping, the checkup.
Dave Schappell: [00:22:59] It's, uh, it's amaze. Once you get in there, it's a, it's, it's a place you can get in. You can't get out. 
Josh Petersen: [00:23:05] Another feature, I guess, just to give you another funny. One was one Christmas when things had gotten to a point where we weren't all in the distribution centers, but we are all given a mission to figure out what could you do to, um, reduce customer contacts.
So we all kind of shifted focus for a while and somewhere I, again, can't, can't really remember who this was, although I'll give credit to all the web devs in a way, because it only required their work as they just added a footer to the bottom of the page that had customer service links in it, and like customer service or self service links and contacts went way down.
And then we ha we already had the real estate. So we started putting instant wrecks in that footer. And it was another huge, huge game. You know, people, people think you want to be at the top of the page, but the bottom is pretty lucrative. 
Dave Schappell: [00:23:55] Especially if it's, uh, something that's attractive to you, whether you know it or not, once you see it, it's, it's, uh, obviously connected to other things that you're interested in.
Um, so before all of these personalized widget let's, I don't want to talk about the, the shopping cart right now. Cause that ones there was probably nothing there before you got there, but, but on the homepage where you'd have a personalized widget or on a detailed page for a product, what was the, how was Amazon making recommendations if they were making them at all?
Or was it purely content blocks that somebody would write? Like, was there any, uh, per I'm not using personalization? Was there any customization or was it all just everybody got the same thing. 
Josh Petersen: [00:24:39] So separate from personalization, widgets. 
Dave Schappell: [00:24:42] Yeah. Like, you know, before that was there. W what I see the same homepage as you, was there anything that would adapt based on our prior shopping experiences or.
Josh Petersen: [00:24:51] Outside of a web lab. I can't, I can't think of what you would have seen. That would be different. You know, our team made a conscious decision to try to avoid customization, to always make the site personalized, based on your experience and your usage of the site. Right. We sometimes built features that were, um, had preferences to them.
But, um, in general, we tried to deduce what you're interested in rather than asking you, cause people miss describe what they're interested in. 
Dave Schappell: [00:25:22] And can you talk about how, how did you do that when you didn't know who the person was? So you know that the new, new customer arrives on the homepage or on, uh, you know, I'm assuming you're trying to learn quickly based on the first item they landed on and what they clicked on, but was how did the algorithm work when there was nothing to know about the customer?
Matt Round: [00:25:45] I think we probably fell back to tell centers. Yeah. And we're trying to get you to rate your visit items, right? 
Josh Petersen: [00:25:52] This ma maybe Matt could think of remembers is better, but what I have a certain point, we changed the team's name to personalization and automation because we realized we were doing a lot of work that wasn't, um, particularly personalized. 
Matt Round: [00:26:08] But it's still valuable. Yeah. And it turns content into live content rather than hand managed content. So for example, new releases and top sellers in any category or subcategory or things that you can, you could compute and always have up to date, you don't need to deploy our most expensive resource are humans who can do the most clever things to figure out what's selling most and then list them out in a one, two and three.
So we really want to get humans doing the unique and creative things humans can do and remove as much as possible of the automateable work of building out the website. Yeah. And I think one thing.
Josh Petersen: [00:26:43] I think it's more Matt, Sarah, or at least where I got to, but really important innovation. It's almost like the next stage of web lab was king of the hill algorithms. Do I say anything about those? 
Matt Round: [00:26:54] Um, sure. Well, I guess the, the basic idea is, um, I think it comes from the observation that data trumps intuition and that we often think we know what the best thing to do is going to be, but we actually don't and customer behavior is what's going to guide us about that.
And so rather than assuming the best thing to be at the top of the page is always top sellers And then it ought to be new releases next. And after new releases, we should put the new Harry Potter book because it's the new Harry Potter book. what we did instead is set up a system to try out different, competitors and, let whoever performs the best stay king of the hill.
it's a bit like a kind of continuous Weblab not as, effective and not as fair as a real weblab because you can't do that true kind of separation between people or measure the genuine impact, but in that sort of organization. So it's trying to figure out automatically what's the right thing here, because.
in some book categories, new release is going to be the most important thing by far in other book categories. It's not the new releases that are important. It's the staple top sellers And, you know, we can, we can automatically figure out those sorts of questions. 
Dave Schappell: [00:28:02] Right? So things like hit, you know, biographies or hit me historical or literature. It might not be new releases. It might be bestsellers just because that's what people are there. 
Matt Round: [00:28:11] Yeah. Shakespeare should be at the top of this. Right. 
Josh Petersen: [00:28:15] Um, so if you, if you think about it in one way, you could, you could have all that technology working and not really have a personalized store for every customer. Right. But if you're ever going to be able to build out the store for every customer, you have to have slots available, not, not predetermined. And I think part of it, if, you know, you have to think of how early this all was because, you know, 20 years later, it's like, we're describing. I mean, anybody would be like, so it's like an ad server.
Right. And it's like, yeah, that it is like an ad server where all these slots are ads, but we didn't know what we were building in that sense. Yeah. 
Dave Schappell: [00:28:50] So Josh, you started as an editor, right? Am I right about that? So like, how did, how did your experience working and maybe talk a little bit about how did the editorial function work before any of this personalization and automation really existed?
And like, how did that transition over time? You know, with sort of editors controlling all the content to review content and you know, they they're still editor, so it's just, but I'm assuming the personalization personalized, widgets and slots started taking over more of that as, uh, so that every page could be personalized as much to the greatest extent possible.
Josh Petersen: [00:29:28] Well, I think when, you know, when I started the site was almost all search and then, and homepage and detailed page, not that was the, that was the entire. Amazon product from a, you know, what we think of the product today. And, uh, the editorial was really important because it was both the voice of a very early product and to help define the brand.
Um, Amazon, when I started to was, was the New York review of books all on sale, uh, 30 to 40% off. And it was really how Amazon group grabbed a ton of attention early on. And so it's, so it's almost it's affinity with literature. Literary criticism was there from the start and had a group of really amazing editors.
And I'm think of, um, you know, James Marcus and Todd Nelson and, uh, very Trinkle and Jonathan tremor. And it went on from there. And so it was, it was sort of hand selling and giving a sense of, uh, authority to what Amazon was doing. Um, and I started writing reviews, book reviews, where my job was to review.
All the nonfiction books that were in the New York times book review each, each week, which is about 15 book reviews to write. And I think I only did my job for about six or seven weeks before I thought this is absolutely going to kill me. Like I can't, I can't do this. My first idea was to hire someone else.
And, uh, I remember in the hiring meeting, Jeff was there and had some input and I said something about, it's just possible to read these books that fast. And his comment was, uh, if anybody's reading the books, they're doing their own thing. And, uh, which was a bit. 
Dave Schappell: [00:31:13] Tell, tell the rest of that story. I've never heard this story before.
Josh Petersen: [00:31:17] Yeah, but that was about the end of the story. It kind of gave me a sense. I guess two things was like, this is not a scalable model to have humans writing these book reviews. Although some of those humans came up with some really invented ways to write the scale for awhile. And the other part was, uh, I mean, I, I didn't see it at the time, but Jeff obviously had bigger ambitions than, than reviewing all the books in the New York times book review.
Like that might've been our important mission, but that wasn't really the mission he was most concerned about. Right. Um, so I don't want to, you know, without making it nauseatingly long, it became clear. We were going to have to use the same kind of software that was powering search and the catalog to get content out in front of people, you know, be it, um, lists of new releases and top sellers, or be it using the catalog data to build out brows and, um, letting editors specialize in their own browse areas.
And, um, and at that point you have slots of contents. Right competition for slots of content. 
Dave Schappell: [00:32:22] And, and when, uh, what we now know of as customer reviews and user generated content, that sort of thing is that was, was that testing more being done by the community team of servicing? I'll just say a user review of a book versus an editor review of a book.
Was that, was that more on that team? Cause we don't have to explore it here or do you, is that not a recollection because I just remember those tests going on to say, do customers notice that, you know, can they distinguish the difference? 
Josh Petersen: [00:32:54] Well, I, I certainly remember JJ running with that, um, mission from the beginning toward the end. Yeah. And, and finding she was quite bold. I thought to find the value of a customer review and to do it, she had to take reviews down on off and try to measure it that way. Right. I guess I don't think I have many opines about the present Amazon in this way, but boy, do I think. The state of customer reviews on Amazon is one of the big misses.
You know, it was such a valuable piece of Amazon and it's become, you know, more of a liability in a way, because. 
Dave Schappell: [00:33:31] Nowadays you need a little bit of a PhD to know how to sort it down to verified reviews and read through sort of obvious, you know, uh, paid reviews and things like that. But, um. 
Josh Petersen: [00:33:42] Such as target of scams too.
Dave Schappell: [00:33:43] Yeah. Yeah. Well, I'd like to think somebody's working on it, but, uh, yeah. So, uh, so we talked about web labs and, and, and I'm assuming the short version of this though, is we just ran hundreds or thousands of web labs and just, and the personalization team as it moved from you to Matt and sort of talk about more how those, how those, it just started taking over more and more real estate, like is, is your feeling that the majority of Amazon's website right now is.
Uh, what's the word own driven, you know, powered by personalization. Is that fair to say, or is it gotten a lot more robust than that? 
Josh Petersen: [00:34:28] Oh, what do you think that I have my opinion. 
Matt Round: [00:34:31] Um, I last worked at Amazon, um, 11 or so years ago. And at that stage, um, certainly a large portion of the site was automated or personalized.
Um, I've not been really watching carefully how the site has developed since then. Um, but I know that there's been a lot of attention, particularly around, um, external advertising. Um, and, uh, I wonder if that's kind of dreamed some of the ability to continue focus resources on the retail side, I guess, with the explosion of web services and other business areas.
I mean, it seems sometimes the retail is not as significant to the company, um, as it once was. 
Josh Petersen: [00:35:09] Yeah. I think I've got a chance to work with the personalization group a couple of times and just the last five years, I guess. And, um, great, great teams, really strong teams, great leaders. Um, I worked with Srikant and Dan Lloyd, who are both still leading the charge there.
And, um, I would say too, that you're thinking of like, what I think is the best big feature they've come up with is the Buy it again, it's the focus on replenishment and helping you if you talk about a store for every customer today, it's a lot about reordering and Finding your old size and things like that.
But then the personalization team has done all that. absolutely advertising has taken over and in a way it's that same slot filling for, dollars piece, and it takes away from a personalized experience. And then I think too, Amazon's always used it's big real estate. I think it believes when it has to launch Alexa, it's going to own the home page for the next 18 months.
No, it wouldn't. when it spent all this money on prime video, we're going to, put it on the page, whether it works or not. So I think sometimes there's just a will to not measure. 
Matt Round: [00:36:21] Or a kind of choice to make a marketing spend. Sure. We understand this is not going to sell product in the same way as personalized content would, but the company has some, larger strategic goal that were willing to invest marketing dollars in.
Dave Schappell: [00:36:35] Yeah. I sort of feel like it's, whether it's prime sign-ups or it's just, you know, video views or, you know, whatever the goal is. I, it's not as easy, uh, easily determined or evaluated versus current sales or, you know, I agree with that. And 
Josh Petersen: [00:36:52] I think just vision is totally relevant that store for every customer. Yeah. I don't know that that's a priority. Um, even, you know, I think the presentation team is focused on sales rather than maybe that yeah. That vision.
Matt Round: [00:37:06] It is fundamentally a remarkable opportunity. You know, if you think about the physical store analog. Um, you can lay out the physical store, but only once.
And everyone has to walk into the same store of being able to completely, they have the store individually for every single customer as they arrive. It's just a remarkable opportunity. The kind of digital freedom you have in a website is, is an amazing kind of venue for creativity. Lets you do things you could never do in the physical world.
Dave Schappell: [00:37:31] Yeah. I look forward to the magic. Like I just feel like too often you go back and you're just seeing what you were there shopping for last time. And I don't know, there's gotta be magic to it somewhere to make it better than that, but it's almost always like, it feels like it's, uh, it's a one shopping episode too late with my current recommendations, you know, versus being more anticipatory, which would be very difficult to do, you know, without showing me totally unrelated things.
But um, but it's the type of things I. You know, I'd like to think the two of you would be working on if you were still working, because you agree with me. Like it's almost every time I go to the page, I'm looking at what I just bought and, um, it's not terrible. Yeah. 
Josh Petersen: [00:38:14] There's a sad effectiveness of retargeting. And we see this all around the entire internet. 
Dave Schappell: [00:38:19] It follows you everywhere. 
Matt Round: [00:38:21] It's there because it works. Unfortunately, that's right. 
Josh Petersen: [00:38:24] When I was back in working on personalization, you know, five years ago, six years ago, we just, we called it the hall of mirrors that everywhere you turn, you're just seeing what you had just done before.
And part of the problem with measuring it this way is that you can't, you often can't beat the retargeting, right. But you have to have that vision that you're going to make some something else. I always think, you know, when we were working on all this mobile experiences were basically non-existent and you know, I think your phone is the best store for every customer you could hope for and all that.
Cause you've got it with, you knows everything about you in these different ways. And, um, You could really build like an effective reordering system without having to run the, uh, front page. Right? 
Dave Schappell: [00:39:09] So a bunch of the things we've talked about is how the team changed. And I wanted to ask about, um, there's a concept a lot of people have heard about it's called two pizza teams and it always felt like the personalization team that you created Josh, was it, it always felt to me like that was the first two pizza team.
It may not have been called that. Can, can you just maybe tell people maybe to do the step back, talk about what a two pizza team is, and then answer the, like, say, did that come from, from personalization or the, the teams that the two of you built? 
Josh Petersen: [00:39:41] What was sort of built under my watch, but I think two pizza teams, uh, emerge more under mats, but I do think we had a model for, but we had a model for, it was a team that could own its whole space.
So I didn't need to go borrow resources from other teams. And, and we did that by. Hiring designers. We're the first team to ever have a designer on a team that wasn't in the design or, um, making web dos part of the whole dev process, um, instead of sort of a separate piece there. And I think we really puzzled, uh, developers by having program managers and TPMS who sat there and kink, you know, worked right next to them.
Um, but that it helped us do so many more launches than a typical team could at a completely different rate, um, which was really important to our success because we didn't know what was going to work. And we really had to chew through quite a list of ideas to find that out. We wouldn't have been able to do it without already having those people on the team.
And, um, I do think if it made an impact among, you know, other leaders in the company that this model had merit rather than being organized by product lines or, uh, right disciplines. And, uh, I would say that maybe the. Um, Matt, you want to talk about what, what you saw evolve? 
Matt Round: [00:41:00] Um, well I think really that the Josh Josh's cross-functional personalization was the model for a two pizza team, just like a bit more like a four pizza team than two pizza team.
Um, but that, that idea that you've go all the functions together, essentially it's like a, a tiny startup inside the company that has the ability to go from soup to nuts on everything and the benefits from the strengths of all the different functions. So, you know, I think, um, the, the impact of having designers inside the team was, was huge in terms of our ability to prototype effectively and to envision what our product was actually gonna look like.
So I think the, the, the success of personalization and shipping product, um, was something that Jeff and others, um, wanting to see if they could. If they could capitalize on. And that led to this idea of small cross-functional teams with a narrow focus, with deep expertise who are going after one thing, and you aren't going to be continuously pulled sideways by giant projects, because I guess one of the big things that was happening in the, in the personalization era still was big.
NPS would come in on big NPI, the new project introduction system, which ran the kind of giant cost company projects. Things like we're going to build toys R us a website would come in and they'd want attacks from every organization in the company. And, uh, trying to push that way, trying to enable teams to operate independently, uh, was kind of what was at the heart of it.
And so working, um, with definitely his direction and taking, um, Josh's personalization team, we reorganized it into, I think it was four, two pizza teams initially. So we narrowed the focus even more, which I think was quite. Where are you at the time? Because the freedom of personalization in the scope, it's one of the things that makes it fascinating and interesting, but we narrowed the focus even more into four sub units, gave each unit, um, resources, um, all the way across the kind of, um, functions so that they could do everything they needed to do.
Dave Schappell: [00:42:57] Can you, can you talk about like roughly what those four, you don't have to remember them all, but like what, how would you break down? How did you break down personalization as an Oregon to four separate, you know, wholly contained or any one or two of them that jumped out? 
Matt Round: [00:43:10] Um, so we, uh, we had personalized merchandising and automated merchandising kind of breaking into those two functions, you know, which is the data-driven automation, non-customer centric versus customer centric.
Right. I think we had an inline messaging team. If you remember, there was that little Stripe underneath the, um, tabs, uh, and, um, we were trying to display things in there that would do useful things. And I think we launched automated advertising really early on as well. So I think those might have been the first.
For two pizza teams or maybe there's Amabel actually, maybe it was  was the other team we launched? 
I think we acquired it, um, just after, just after that initial reorganization. Um, uh, and the idea with the two pizza teams is so they're largely independent. They can do their work without calling on other people. Um, they're allowed to proceed independently and they had what was called a fitness function.
So they had this kind of mathematical computation that said whether you're winning or not. And they had a long-term goal focus for the long-term, I'm driving up this fitness function and we spend a great deal of time trying to think about what was the right way to measure personalized merchandising.
Um, and, uh, I think it was something like, oh, I can't even remember. It was phenomenally complicated function that we ended up with is how well we did with the number of opportunities we got compared to non-personalized content. Um, or something like that. And the idea was you just watch your fitness function on a graph, and if it goes up you're winning and everyone stays off your back.
And if it doesn't go up, you're losing and maybe you're finished, or maybe you're just not making, um, the, the, the right decisions. So that was kind of the theory. I think we had a lot of trouble really making, um, fitness functions perform in the way we wanted them to. Um, I remember Jeff being we'd finished the discussion on personalized merchandising's fitness function.
Um, Jeff thought it was a, a good piece. I thought it was a good piece, a great piece of work. Um, Brendan worked sprint Smith ran that, um, two pizza teams to work really hard on it. It was a great idea, but I think ultimately it turned out to be quite hard to measure and drive. Um, even though it was tightly measuring what the team was focused on.
Dave Schappell: [00:45:18] Yeah. It always felt like the, the idea of two pizza teams was really. Uh, uh, easy to understand, and it was effective. And I think Jeff really liked it because of the speed. But the one thing that never seemed to work was fitness functions as well-intended, but very difficult to, um, to come up with them.
Like I think I actually interviewed Blake shul for talking about automated advertising. I think his was probably an example where it did work. Right? Because it was very focused. We're going to run these ads on Google. How many are we running? And are they driving incremental sales? Are they profitable?
But a lot of the other ones were probably just a lot harder to do with a real-time function versus web labs and things like that, which are more test oriented. 
Matt Round: [00:46:01] Yeah. Yeah. 
Josh Petersen: [00:46:03] I think anything focused on optimizing is going to work better in that space. I think anything focused on innovation is a little bit hard to even articulate, right?
Dave Schappell: [00:46:12] Because you're testing brand new things and you're not really fine tuning them. You're just trying something. 
Josh Petersen: [00:46:17] I think too, the idea of these there's a bit of a. Confusion, maybe about what a hardened API would look like when you're working with teams of people and, um, and how many there were things like that.
So I, anyway, I definitely saw us be able to get a lot done by being able to draw on our own team. Right. And sometimes once everybody was focused on, I mean, Matt could speak more to this than I can, but I just have heard, had heard more, I guess, about once you have your focus, there's more narrow focus.
You're a little more limited on how you can collaborate, right? Yeah. I'm sure there must be a tension between kind of wider collaboration and narrow focus and, and, and, um, I wonder whether the, the earlier personalization team was, uh, uh, a better size. Um, but yeah, it's hard to, hard to know, but it's either way they both feel more like a startup.
Yeah. Okay. So you couldn't have done lots of those. Yeah. Yeah. 
Dave Schappell: [00:47:11] So we're at 47 minutes. I have two or three things I want to ask. One is cause Matt Mattson, Scotland, and haven't seen him in 20 years and I've always wanted to ask about, uh, MRT. This is a little I'll I'll run through, and then you guys can answer them in whatever the right order is.
Um, somebody queued us up with, uh, saying that, uh, Matt should tell us about a ketose if I'm pronouncing that correctly and sort of how it ties to, um, what we're talking about. And then one of you just mentioned AMA buck. So those are my three things I wanted to ask about. I think they're all somewhat related to this or, uh, interesting to listeners.
So, uh, which order does that make sense? 
Matt Round: [00:47:49] Um, all in 10 minutes, um, MRT, uh, was the, the internal name for Matt's recruiting tool. And that was just a, an attempt to. Put some software and some process, um, around, we were trying to hire a lot of people, a vast number of people, and it was hugely expensive and complex to organize and make it work while our recruiters were great, uh, working, uh, working their tails off, trying to get things done, but they didn't have particularly good tool support.
And I guess my observation, there is areas that are less exciting to technical people tend to have less well developed kind of systems and tools. So we used to use PeopleSoft, I think for recruiting and it was not a particularly great product from Oracle at the time. Um, so I just built a very simple tool to track feedback and where candidates were up to in the interview loop and, um, try and make that run more smoothly.
And, uh, and it turned out to be somewhat useful. 
Dave Schappell: [00:48:47] Somewhat useful. You arguably like built that over that the myth is like a weekend. Is that true? 
Matt Round: [00:48:52] That isn't, that's not really true. So I mean, the first person that they warrantied was very. Poor. Um, my parole is still terrible. And so it was, it was an evil piece of puddle and it did very little.
And we grew incrementally, I guess is probably one of the learnings from personalization's approach is you don't want to do big design upfront grand product launch, but we launched a minimum product, which barely did anything. And then gradually added features to that over time, which made it slightly more useful.
Um, so it gradually developed and, and frankly, I probably spent more time writing software than I should have done as a director because I really liked making things myself still. And I really kind of missed that aspect of my work. So I think, I think probably I suck more director time in there and less weekend time in there that the story would tell.
Dave Schappell: [00:49:37] Well, I, I would just share, I mean, that, that, and again, people worked on it after you launched it, but that thing lost, it lasted forever. Like it, it felt to me like if they might still be using a version of it, like when I went back in 2012, it sure looked a lot like MRT still. So for, you know, for a company that was hiring thousands and tens of thousands of people, and it's a, you know, it's a great example of somebody built something because what we were doing before MRT was horrendous, like literally emailing resumes around to people and it was, it was out of control.
So, um, it's a good story to tell. And now I know it's Matt's report and recruiting tool. I had three different, uh, three different descriptions. 
Matt Round: [00:50:19] And other tools as well. So. 
Dave Schappell: [00:50:22] That's the only one you'll be remembered for. 
Matt Round: [00:50:24] Absolutely. And then just, you want to talk about. 
Josh Petersen: [00:50:26] Sure. I am. I, um, I might need a big piece of clarity here is this implementation of king of the hill, right?
Is that right? And, um, so we had this obviously controversial way to influence the website. So we picked our, one of our most diplomatic program managers to work on it. She's a Russell Dicker who, who himself has a hardened API, but no Russell Russell. Who's really great kind, man. Um, not at work, uh, um, for the Bronx of what we, as a team chose to do.
And, um, and he had to go explain to everybody how essentially the entire site lost the placements they thought they owned. Um, and then now we'd all be competing and, um, something like that. And, uh, it particularly impacted product homepages that that thought they were, it was their product. And so they would, it was there, you know, it was the toys home page, so that toys team would decide where I'd be on it, understand, I mean, understandable.
Um, and, um, It didn't win tons of friends among, um, editorial, their product people when that rolled out. It did famously, I think, you know, David famously did earn Russell a, a Valentine in the, um, the stranger. 
Dave Schappell: [00:51:48] And I'll put, I'll put it in the post. It reads dearest AMA bot. If you only had a heart to absorb our hatred, thanks for nothing. You jury rigged rust bucket, the gorgeous messiness of flesh and blood will prevail. 
Josh Petersen: [00:52:03] Yeah. It's, it's cute. You can tell it came from editors. 
Dave Schappell: [00:52:07] It's well-written. 
Josh Petersen: [00:52:08] And, uh, you know, we, we had our own, um, literary reference on this is, uh, at a sign in my office that said, um, people forget that John Henry dies in the end.
There's this, the, this folk tale of man against steam engine steam driver. And, um, It's just hard to scale something like Amazon, without always looking for a technical solution. You know, it's hard to think of any way you can put people at that. And, um, it, I think it was a mistake to think we didn't care about the people, right.
And certainly working in editorial or I've worked in retail for eight years before that, you know, I really believe that. And selling and getting the right item in front of people, which is the technology we felt, but change, you know, none of us understood the scale of what we were building. Um, and, uh, there was not always a good product and I'm sure it was really hard if you were trying to put handpicked content to understand that that wasn't going to last.
Dave Schappell: [00:53:12] Yeah. Well, I, I mean, I, I, I know that, yeah, you didn't have a lot of fans for this at the rollout. I mean, the only thing I would say from being 20 years older now is I do kind of feel like Jeff and others should have. Owned more of that verse. You know what I mean? That that are not argument, but you know what I mean?
The disputes that were going on, like I felt like there were a few months, this is from my sitting at another desk. I felt there were a few months where it was like letting people, duke it out versus hearing that this is we've got to do this and we got to test it because the company depends on it. We can't, you know, it's too expensive to do the other way.
And we're going to be adding another, a hundred categories and sub you know, all this sort of stuff. And, um, and I think a lot of that was left to the devices of Russel, uh, to, to communicate with, with all the other teams. And, you know, I don't know that that was always my, uh, uh, impression of what happened.
Josh Petersen: [00:54:06] It's worth pointing out. We had, we did have Daniel spills on the team largely to smooth people's feelings and communicate in meetings that the rest of us didn't really want to go to. And, um, he, he was like the soft API. Yeah. We had good cop, bad cop on the team. And he was the fixer. There was that. I mean, that's definitely part of the story early on.
It's a hard part of Amazon's growth, right. Was to, um, make that change. And, uh, I don't know. We didn't always do it very well. I would say that we didn't get it with some cavalier spirit. 
Dave Schappell: [00:54:44] All right. Fair enough. And I, I know I'm not leaving much time by just mad. I fear I'm not going to get you on a call again. So just love to hear the ketos story just because it isn't, it's an early Amazon web service story and there's, there's a bunch of them. We already had a Reuben telling him, and now you're going to say no, Ruben. I actually had the F no, go ahead. 
Matt Round: [00:55:04] Ruben Ruben had the first Ruben had the first. So, um, I guess the issue we were dealing with at the time was the, the website system architecture.
Uh, was quite limiting in terms of what we could do. We have, um, large volume data stored in backend databases, but they couldn't sustain massive query rates. And then we had small volume data kind of deployed out to the front facing, um, onlines. And the small volume data was causing us pain because we were trying to do recommendations on the homepage.
And we can only store your last eight purchases because eight purchases times, and, and I have a many million customers. It was at the time equals this much storage space with the machine has only got room for a few of those. So we kept on running into the problem where the recommendations would recommend things you'd already bought and everyone would say stupid robot.
Why did you recommend things already bought is because our data was limited. So there was a project we were running called eight is not enough to try and understand how we could get more data available more quickly. And, um, Rubin at this time had just created the search service essentially by duplicating one of the front end onlines and running it purely, um, as, as search.
But it was largely kind of the same thing as the, that was running, uh, at the front. And, um, I was toying around with whether we could make something smaller and simpler. And the first thing I built was a little kind of remote access, a BDB on another machine across the network and come back. And it was a big surprise to me, really that that was, I didn't think it was going to be fast enough, but it was real, really very performing work, surprisingly well, and I'm sorry, I started, uh, with, um, Val, uh, golf men, Andy Harvick, and Brent Smith, just thinking about generalizing, um, uh, kind of, uh, ability to make a remote function, call to another machine and then get data back.
And, um, this is like 1999, 2000, probably 2001 working on this. And, um, we discovered we could do. Uh, we could enable new features like click stream became possible because of this. We were able to store every customer's, um, last day items they had visited, uh, which is a huge amount of data, but we could do that because it could be separated from these onlines that were doing the principal work and clickstream enabled all sorts of features on the website.
Uh, we discovered we could access all of your purchases and ratings quite quickly, um, on the front ends or even we could move computation like computing your recommendations from the front end machines that were overburdened to kind of second tier machines. And so we built out a generalized architecture for sending messages to another machine and getting responses back.
And by 2001, um, we were, we were in a position to invite other people to kind of build on top of that. Um, that framework and so that lets you move data away and it lets you move computer way, which I guess in some ways, like I genuinely did not think, um, as big as Jeff thought at any point, but that is S3 and easy to at the end of the day, I didn't see it.
I thought webs, you know, um, if the early days of web services, I thought it was a terrible idea that we were giving away our best data. Why would you give anyone? Our similarities data are crucial, crown jewels. Why would you just give that way? I thought it was just a dreadful idea. Um, so, um, vastly wrong on all manner of things.
I had no foresight about that product, I think. 
Josh Petersen: [00:58:27] Yeah. To say something about how the multi-disciplinary part of  worked, if there's a good story to this too, because, um, you know, the PMs and others were sitting around going, this is laying where we're recommending stuff. People already own, you know, the first request to the SDS on our teams, that's really hard, you know, tough, but it was, we pushed it.
Or you've just heard all the genius bit that Matt and, and Val and Greg and Andy were able to generate by trying to answer this other originating requests for like not being, not having a crappy product. Right. Well, plus we had the multi-disciplinary team to then call the project aid is not enough, but to make all the difference when you've got a good name for a project.
Absolutely. And, and, you know, the, the kind of pressure from the product side produced innovation on the technical side, the innovation on the technical side then enabled a whole new suite of creativity and open doors. You know, the, well, some that we saw right away and some that we really had not had not imagined.
Right. And so it showed us the leadership too, because honestly on the product side, people were like, why do we care about this service? And you just get us more, more, better data, but it took their insight. You had to build the right, the right solution. 
Dave Schappell: [00:59:42] And to wrap that. And so with this, that was called a ketose. Was that the name of the. 
Matt Round: [00:59:46] The kind of service framework? Yeah. Yeah. And we deployed quite a lot of services on top of it quite quickly. So we had your purchases, your ratings and your click stream available remotely with high performance, um, really quite quickly. 
Dave Schappell: [01:00:02] That's awesome. Well, I like to wrap every episode, just asking sort of stepping back from all of those details, you know, and, and all the specific, uh, product launches or features launches.
What do you think the lessons are? You know, if you think about, uh, for entrepreneurs coming out, what would you, what do you think about your time working on this stuff with Amazon, that sort of more timeless things that you would tell people to, to think about, or, or if you're advising a startup or an entrepreneur who's working on problems like this?
Josh Petersen: [01:00:33] The thing I think about is that you don't actually know what's going to work, so you have to have a framework for getting things out quickly. Measuring them, uh, doubling down. If they look like they're going to work when moving on, they look like they aren't. So I don't, I think it's more common that people think we've got this one great idea.
And we're going to make that happen where as it might be just more efficient at work through five and then refined. Um, so I think that was one of our real successes is that we, we tried a lot of things and, um, learned about a lot of things we didn't know were going to happen. 
Dave Schappell: [01:01:07] Hey, Greg, Greg Linden wrote something similar. He said one recurring theme was trying something. If it works double down, if it didn't work, don't be afraid to try a variation. And sometimes you just double down, double down, double down, and that that's sort of the story of the multiple generations.
Josh Petersen: [01:01:24] To think that your first ideas aren't maybe going to be as good as your sixth, seventh, and eighth.
So you have to find a way to work in a varied way. I think could we kind of, we used a budgeting system that almost made forced us to work in areas we weren't necessarily. Uh, we thought they were promising, but not clear, but we only took about one of those ideas a quarter. And by three that we thought would work.
Matt Round: [01:01:48] Brian so mean this was operating in a very agile way. Uh, and now, now agile software development is kind of, you know, it's the normal way everyone to think about it, but, but it wasn't, uh, we were in a world of big design upfront. Imagine the project plans six years out, you know, we'll, we'll walk through this huge thing.
We know what the customer wants. And I think some of the personalization revolution was, you know, um, minimal product, bring it to the customer. Does it work and then iterating on that quippy. So Josh has quarterly planning, um, just kept us moving into new ideas all the time. I think it was a really helpful process.
I think you can't underestimate the value of the cross-functional team. Um, it was amazing how much more we could do when we put the business and the design, the product and the technology together. Um, th that was, uh, uh, remarkably more powerful than technologists get stuck up on, you know, they're, they build things that are interesting to them.
Um, if you're surely in the business side, then you, you can understand sometimes what's technically impossible and what's technically easy and there might be quite close to each other as far as you're concerned. So to putting the different functions together is huge. And then I would say, um, data trumps intuition is another big learning.
I was wrong about so many things. My, my hunches and my intuition, what I thought would work, what I thought would make a positive or a negative difference. I was wrong time and time again. And, um, listening to the data was really important. 
Dave Schappell: [01:03:12] Yeah. It's yeah. I, the one thing I will say again, as a outsider, observing, like, it always felt when I, when we did learn about scrum, which is a, uh, product development methodology, if you will, when I first read about it and I'm like, Hey, it fit my personality much better than doing waterfall project work.
But I thought about the personalization team, like, it just felt like things were going. Fast, you know what I mean? And, and my, and I wasn't in the room ever when you were presenting to Jeff, but my hunch is that that's what he loved about it. That there were just a ton of experiments going on.
Everything's being measured and it's, and there's no, there's nobody saying this has to be done this way. It was almost all like, when you think about the people that get hired, who were experts at things they didn't do that well often at, at Amazon. But whereas people that were saying, let's look at the problem, let's look at the customer problem and how do we solve it better.
And T and run lots of tests. Like that's sorta in my mind, that was when I thought about scrum. I thought about the personalization team. And that was years before, at least I think, I think it was years before scrum existed. 
Josh Petersen: [01:04:19] We had our own version, but what I would say too, is it wasn't like a, it wasn't like a development methodology.
It was like a product management overview or something like that. I mean, I guess that's how I slept. Like we weren't. Giving, you know, small tasks to Matt or Andy or, you know, anyone, it was more like, uh, this is the theme for the month that you're gonna run with. We almost did two pizza teams by the quarter.
Right. And, um, but what, the way I was, I read something somewhere that just kind of blew my mind. It's basically like the information is out in the future, right. Six months from now, you'll know what will work. And so if you think about it building a prototype and so final design, you can pull information toward the present.
Um, cause it's gonna take less time to get that information. And, uh, that's a lot of how I looked at it as like, how can we learn the most, the fastest, um, by looking at, you know, what can we do learn the most of the baskets. Right. Does that make sense rather than. 
Dave Schappell: [01:05:18] Yeah. We run as many run, as many run, as many tests as you can. And let people tell you versus arguing about what's going to, you know, what what's going to be perfect, you know. 
Josh Petersen: [01:05:28] Before testing for testing, say, yeah, I always thought it was pulling in the information, like how can we. 
Matt Round: [01:05:35] And we just had a lot of fun as well, to be fair. We just had a lot of fun. You know, Amazon used to talk about work hard, have fun, make history.
And I think, um, I think personalization had an amazing opportunity to do, to do all of those 51 0 3, the big room we used to call it, um, was, uh, my happy place. 
Josh Petersen: [01:05:51] It was yeah. High point of my career to work on with that team that we, you know, we felt and honored to work with Matt and Greg Linden. And, um, that was as leaders on the team Branson. Um, others, Daniel, Andre, just, just doing a really star team. 
Dave Schappell: [01:06:09] Awesome. Well, thank you both. Uh, so, uh, Mattson Scotland, Josh is in Seattle. This episode only took about six weeks to get scheduled on the calendar. Uh, but thank you both so much for being guests. I, this every, you know, everybody's talked about personalization, I think in every single episode.
So I think this was great. And I look forward to talking with the community team as well, because there's a whole basically complimentary set of, of, uh, features and functionality. So I personally found it really interesting as you can see, I'm asking questions about things that I actually didn't hear the stories before.
So I think other people are gonna like it too. And, uh, and as I've said to a bunch of people, it's just great to see you both. Really good to see you. Thanks for making the opportunity for the audience. Thank you for listening to the event like an owner podcast. If you'd like more details about what we discussed today, or want to contact me with that, it's suggest future guests or topics, please visit invent like an owner.com to sign up for the weekly newsletter.
And that's it for today. Remember? No sniffles. .