Eric Benson: [00:00:00] So we moved from the, uh, Soto office to, uh, the, the Columbia building downtown. And that was also not a very fancy building, but, uh, that the, uh, the landlords really didn't want to have any pets in there. 
Dave Schappell: [00:00:19] It was awful. They had no right band. 
Eric Benson: [00:00:23] But yeah, but they really didn't want to have pets in there. And there, you know, fears of liability if someone got killed and you know, those, those things.
So, um, yeah, the landlords asked Amazon to sign a waiver. Or to hold them harmless, if anything happened because of the dog. Now, the funny thing is then Jeff asked us as Susan and I to say,
Dave Schappell: [00:00:57] this episode is sponsored by rebellion. Defense rebellion was born to put the power of cutting edge software and artificial intelligence machine learning into the hands of those who protect and defend the United States, the United Kingdom and our allies in this software centric, national security era to learn more, visit rebellion defense.com.
Today I'm excited to be talking with Eric Benson, who was one of Amazon's earliest software engineer. In earlier podcast interviews, his name came up a lot, uh, as he played a part in or drove, many of Amazon's, uh, most impactful features, Eric and his wife, Susan, who led Amazon's editorial team and their dog Rufus, uh, were three of the favorites amongst early Amazonians, uh, for not only their contributions at work.
Uh, but also because they, uh, all three of them had very warm personalities. Uh, Eric's going to share several stories about what it was like building and inventing things from scratch in the Mets, in the midst of insane customer growth, um, and how staying focused on making customers happy, which might just simple as that, uh, focusing on making customers happy, enabled the team to keep moving forward during that chaos.
Eric Benson: [00:02:09] Thanks, Dave. It's good to see you. 
Dave Schappell: [00:02:11] Nice to see you. Uh, so you joined in 96, uh, which was very early. Uh, do you know approximately what Ember engineer you were just to put things in perspective engineers. 1, 2, 3, I think I was maybe five, four or five. Yeah. Got it. So very early. Yeah. And so today, uh, we're going to talk about, uh, several early software engineering projects that you led or participated in, but I'd like to start with the star of the story of Amazon's multi multi-day outage in 1997, because I don't think that's been mentioned yet.
It was supposedly very bad. Do you remember anything about that? Or what can you tell me? 
Eric Benson: [00:02:53] It was in early 97. And so it was about a year after I started more or less. And, um, I recall it was several days while, like you say, I think about three days, but, uh, um, what I recall is that we had a hard database crash that our database server crashed such that we could not run the database and therefore we couldn't run the store.
And, uh, we had. Oracle engineers on site trying to solve the problem. And, uh, it was, uh, you know, we were, we were sweating and, uh, you know, I think a lot of, uh, internet sites in the early days had those kinds of troubles, but, uh, it was, it was the longest outage that we had, I think ever once we. 
Dave Schappell: [00:03:43] So this was basically customers got a 4 0 4.
Eric Benson: [00:03:47] Yes, yes, yes. Come back later. Yeah. Right, exactly. 
Dave Schappell: [00:03:51] Right. And, uh, did it, I mean, in those days, did it threaten the company, like what would have happened if like, I'm assuming we recovered everything. Cause it would be a different story that we would have all heard about if they had to restart from scratch.
Eric Benson: [00:04:04] But yeah, I can't say for sure that we recovered everything, but we did recover, um, at least. Most probably close to all of the data that we had. Um, I know, I know personally because my, my order history goes back before that, that outage. So, yeah. 
Dave Schappell: [00:04:23] That's funny. I lost mine because some, at some point, uh, there was a point in the old days, you probably remember where you could have multiple accounts with the same email address passwords.
Yeah. And at some point they got rid of that. So I lost the first few years of my history. Yeah. Yeah. Um, so you, so you didn't participate in fixing it, you were, you were there experiencing it? Um. 
Eric Benson: [00:04:45] Correct. I was a spectator, uh, I think Bob , who was, uh, one of our w he was, he had sort of adopted learning about the database after he had been there for a little while.
And so he was working with the Oracle engineers. They're trying to get things back on them. Right, but. 
Dave Schappell: [00:05:06] I mean, it's interesting cause I know Eve they're famous stories about eBay having outages even later than that. Um, I mean the only question I have for it is where was it not common practice to have backups of things like that back then?
Is this a silly question or just, was it even more extraordinary? 
Eric Benson: [00:05:23] Not so much a matter of having a backup or not? It's a matter of having a server running. I mean, so we wouldn't have had an entire backup server. We would have maybe backup, uh, tapes or, you know, probably tapes at that time. Uh, but, uh, so that we could restore onto a server from backup tapes, but then you would have to have an entire server ready to go that you could just switch over to.
So no, we didn't have anything like that at the time. 
Dave Schappell: [00:05:51] Cause back then we only had one, one database server. 
Eric Benson: [00:05:54] Um, yes, I'm, I'm pretty sure, at least when we definitely only had one. Customer database at that time. And really that was almost the entire, that was almost everything. There were, there were, there were some auxiliary databases, but that was the only thing that really mattered was the customer database.
Dave Schappell: [00:06:14] I heard similar stories that the eBay crashes were just as drastic, where they could have had to start from scratch and rebuild everything, which for them may, may have been even more devastating because it was later on and they had all those reviews and ratings for all the sellers and, uh, purchase history and all that sort of stuff.
It would have been, uh, difficult. 
Eric Benson: [00:06:33] Yes. They had serious problems. And in fact, I think that was one of the reasons, uh, for our, uh, for Amazon's, uh, drive to do auctions was that we thought E-bay was vulnerable because their technology was so, so Ricky basically unstable. 
Dave Schappell: [00:06:55] And what we weren't aware of is the power of network effects and auctions and a bunch of other things, um, side. 
Eric Benson: [00:07:03] Well, it's a hole, it's a hole. That's where the holes show on its own. But yeah. 
Dave Schappell: [00:07:07] So, so then backtracking, you joined in 96, about a year before that outage. How did you get to Amazon? And I'm not asking this with every guest at this point, but it's interesting because you came with a few, uh, very early and you worked with a few of the very early, uh, employees, uh, before Amazon.
Eric Benson: [00:07:25] So I think, yes. Um, shell captain, who was the first engineer at Amazon and who was mainly responsible for building the, all the, all the, uh, it infrastructure, basically all the software that led to the store opening in June of 95. Um, I was see, I had worked with him at, um, three companies previous to Amazon.
So I w I had worked with him at Lucasfilm for a little while and at, uh, lucid, which was a, a small software company that I had founded, uh, for several years. And then I worked with him at Kaleida labs, which is a, uh, IBM and apple joint venture. And that was just previous to, to Amazon. 
Dave Schappell: [00:08:15] Right. And, and JJ Jacoby was there as well as the other Amazon employee. JJ was an early engineer TPM, I think more so when she joined Amazon. 
Eric Benson: [00:08:24] Yeah. Uh, yeah, well, no, she was a, an engine. She came in as a, um, a manager, I think has a, like a, yeah. 
Dave Schappell: [00:08:35] Like a software software manager. She was a first level manager. Yeah. She's back again. She couldn't get enough. Um, and it was actually Susan, your wife who got recruited and not use that.
Eric Benson: [00:08:46] Well, not exactly. Um, the w the way it happened was, um, shell. Sent an email to people he knew in the bay area saying, oh, Amazon is hiring people. We have these positions available. And at the time I wasn't really looking for anything, but I noticed that one of the positions on, on his email was managing editor and, um, Susan and I, Susan and I were both at that time working at Netscape in mountain view.
And, um, I, I saw that job managing editor. I thought, oh, you know, maybe Susan would, would be interested in this. And I wasn't really thinking of myself at all. Um, but, uh, so I, I showed it to her and it happened, well, that's, this is her story. But, uh, it happened to be on a day when she had some conflict with their boss and, and it would sort of, you know, fill at a time when it seemed like, uh, a good idea.
And, um, I shelved. Uh, tried to recruit me after, after we were up the interviewing for Susan and we kind of decided to maybe both look for positions at Amazon. So, yeah. 
Dave Schappell: [00:10:00] So you got hired as an engineer, like one of the first stories I'm going to ask about his book, Remick book recommendations. Is that one of the, what is that one of the first things you worked on or was it a hotspot. 
Eric Benson: [00:10:10] That was a few months later, but, um, I think from the start, it was just, um, uh, kind of working on a website software, um, not even anything specific, but there were just a bunch of things going that that needed, needed work and, and, um, I can't, you know, I can't even point to anything.
Uh, well, let's see what, um, well, you know, that there were odds and ends even just that there were off to the side, like the, uh, the, um, customer service. The email queue at that time was, had, had grown too from, you know, they like everything else. It had grown from maybe tens of tens of messages to hundreds of messages, to thousands of messages.
And at the time those messages were, um, just stored as single files in a, in a, in an ordinary Unix file system and in one directory in a single directory. And so one of the things I did early on was just to split that up into multiple directories just to, to keep it from crashing the file system. So, um, you know, so there were, there were little things like that.
It's, you know, um, there were challenges due to scaling. There are constant challenges due to scaling. And so there was a lot of just fixing little things that don't even seem like, like projects per se, but they're just to prevent the site. From coming down, but basically the store running. 
Dave Schappell: [00:11:49] And in our pre-interview you mentioned that you never really, or there were times when you felt like you didn't have a boss, but did you report into shell or shell was, did it not, did it not matter as much because it's just like, there were things that needed to get done?
Eric Benson: [00:12:02] Um, yeah, there were, I mean, really the people did have projects that we're working on, but there was a lot of, there were a lot of interruptions, you know, no matter what you were working on, there were always emergencies. So, um, and you know, when I first started that it was still very small and simple, but because of the growth that things just started, started breaking right away.
So, um, yeah. So, um, so I, I got some specific assignments early on, but they were, you know, generally things that had to do with, with scaling. Um, it wasn't until it wasn't until JJ started, then she was my manager working on it. Bookmatch or project. Okay. That was in, so that was actually in 97, I think. So.
Dave Schappell: [00:12:52] Yeah, for listeners we're getting into, uh, sort of, uh, two projects or a project area that you can think of as making better recommendations for customers. Um, so you can suggest books, um, is that, I mean, that's a fair way to, and you, you, you approach it in two different ways, uh, is that right? Like the book matcher was sort of third party software on, and you did a version of your own. 
Eric Benson: [00:13:17] Engine that, that, uh, drove book matcher was, was a third party license, um, from a company called group lands, or at least the product was called group lens.
Um, and, uh, it was basically a way of collecting and, um, matching up recommendations so people could, um, people could rate books and then from based on their ratings, they would get recommendations for other books. Right. And, um, that, that was, uh, became, uh, uh, a project that I was working on to build an interface on top of this engine that would be displayed on the, on the website.
And this was, remember, this was very early, um, website software so that we didn't have things like, like, you know, Java script, it was just strictly HTML pages. So it was very primitive kind of interface. 
Dave Schappell: [00:14:11] But, um, everybody get asked the same 25 books. Like what pretty was like, it was pretty clunky. Like it would be like Shakespeare. 
Eric Benson: [00:14:21] Started out right there was sure, you know, there was like, there would be the most, most popular books in a bunch of different categories.
And then people would go through and rate them. And, uh, it ultimately, it was not a, not a very successful product because it, it wasn't something that people really wanted to use. And it was required an investment of time just to get recommendations, which most people aren't willing to do. So, yeah, it was a pretty big barrier because as you were saying, Hey, fill out this 25 question survey, basically what it turns into exactly.
Exactly. Like, do you have time for a 20 minute survey? It's like, no, I don't really want to do that. Right. Yeah. Only, only people that are curious, you know, basically there are people who are just curious to try things and they'll just go like, oh, let's, let's try this survey, you know, but in general, people won't just do that cause they're looking for a book.
Dave Schappell: [00:15:14] So that was the thing that was powered by group lens, which was a company called net perceptions. And just for historical purposes, it was a, a group out of the university of Minnesota that had come up with it and we licensed it that, and the short version is it didn't work well. And we can talk a little bit about why, right?
Uh, other than just the, the clunky interface where people had to answer 25 questions. So while you were building book matchers, you also built a first version. Was it similar? What did you call it similarities or was, was that simultaneous? 
Eric Benson: [00:15:45] I, I can't remember if it was called similarities to begin with, but it quickly B B was labeled similarities after it launched, but it was just the idea that, um, we would use sales data to generate recommendations for books.
And so it was, it was a simple thing to, um, add it's the, the feature that you see on the detailed pages that says people who bought this item also bought these other items are P readers readers who read this book, also read these other books, that sort of thing. So, um, and it's, it's sort of become a yet at the time.
I think it was when it was new, it caught on, in a way that. Became, um, sort of, uh, something that, that became part of the culture in a way that it was a new thing, you know, that, that people talked about, you know, that there were like new Yorker cartoons based on it. And some things like that. So yeah. He knows me better than my husband that's right, right, right. Yeah. Yeah. 
Dave Schappell: [00:16:47] Um, and so you, you meant I'm going to start, I'm going to butcher this, but you mentioned that, that the one was sort of trying to do, map you to a person and then recommend a book. Whereas this was like mapping many. No.
Eric Benson: [00:17:00] That's pretty much right. It's so that the essence of the group lens engine was, it was trying to build a profile for you, the user, and then match you to the nearest other users that were like you.
And then they could look at the books that, the other users, like you also liked. So it was, matching users to users but this similarities, the way we built it is turned upside down or inside out. But basically it was matching items to items by using the users And so, essentially you would take an item and you would find all the customers who purchase the item and then you would find all the other items that those customers had purchased. And you'd rank those in order of popularity. And there were, there were some modifications to that, um, that were necessary, but ultimately that was the, the essence of the algorithm and it worked surprisingly well.
Dave Schappell: [00:18:09] Right. Yeah. And I'm assuming probably V1 was simply that and probably V 11 had other things in it, like the, what they rated the book after the fact did they? 
Eric Benson: [00:18:20] Yes. There were sure there were lots of, lots of, uh, there was a lot of looking at the results and, and, and sort of tweaking the algorithm to make it more useful or, or, uh, you know, there were some things that were kind of strange, uh, misleading things that happened like, uh, Douglas Adams who wrote the Hitchhiker's guide to the galaxy.
Um, there were, so there were a few books by Douglas Adams and in the early days of the internet, of course, that, that, you know, books like that were more popular among the people who shopped on the internet. But, um, there was another author named Douglas Adams and people who just wanted to buy everything that Douglas Adams wrote would, would buy this books by this other Douglas.
And, and usually return them and not be happy. They basically had made a mistake, but they didn't know it right when they made the purchase. Um, and because we were using sales data, we started, we actually started recommending this other Douglas Adams book on the Hitchhiker's guide to the galaxy, which was, you know, something we had to avoid.
So we had to put in start putting in special cases for things like that, where we could distinguish, you know, cases that, that we didn't want to recommend. 
Dave Schappell: [00:19:35] Yeah. Greg Linden, who was another engineer, uh, gave me some other examples of that later on where they. Use it, I think it was for, um, rec similarities.
Another later version. I forget what it was called, uh, across the site, but like with electronics, they would recommend cables and it would be a bad cable because it got in there somehow people would buy that cable, which would mean it would recommend that cable to more people. So it's a similar problem, but just got extended as these recommendation technologies got spread across the web.
Eric Benson: [00:20:05] Yeah. Yeah. The, the similarities you've had, it's kind of created a feedback loop for some items where it actually increased sales for things that you wouldn't necessarily want to increase Salesforce. Yeah. 
Dave Schappell: [00:20:18] And so why was similarities? I mean, if you had to sum it up, why was similarities that much better than, I mean, I guess bookmatch, or had had the odd stacked against it because first it had.
The difficult to get P impossible to get people to fill out the 25 question survey, but then it's also because it's limit trying to match a person to a person. And everybody's saying they like Shakespeare, you know, I would assume that it would just end up with the same recommendations. Whereas your individual item purchases over time is getting more and more dialed into what you're actually interested in. Not what you say you're interested in. 
Eric Benson: [00:20:52] Uh, yeah, so, uh, a bunch of things, but I think the, the number one reason that bookmatch, or didn't work as well is that there just wasn't enough data and that, because were, there was a lot of sales data. And, um, so we get, we got to use the sales stale sales data was constantly increasing, so, uh, and updated every, every day.
Um, so that was, that was a great source because people had to engage in this activity to, to see the engine. Yeah, there was this overhead that was a barrier to entry. And because of that, there wasn't enough data and there was so it just, you know, yeah. 
Dave Schappell: [00:21:36] So similarities was kind of an early use of big data. I mean, yeah, 
Eric Benson: [00:21:40] I would say yes, it was. I think the first time that, uh, at Amazon we used our sales data, um, basically created a feedback loop where we use the sales data to, uh, promote something on the site. That was the right. And that was sort of the, maybe the first big exempt. I mean, Amazon, first of all, the differentiator from physical is just selection that was available, strong prices, but this was starting to get to that next level where we can do things at a physical store and the people in the stores can't do because they can't possibly have read all the books.
Whereas here you're using the purchase information from all the customers to help find additional books that they'd be interested in buying based on the other purchase. And conceivably, even a physical bookstore could have done that. But I doubt if they had the, the technology in place to do it, but they could have done swipe cards and even just using credit card data.
And, you know, maybe, you know, there would be. Yeah. Yeah. It would have been very advanced in the, in the nineties to try and do that. And the fact that we had that data in a way that was pretty easy to use, ultimately, I mean, there are challenges in keeping it up to date and, um, you know, we, we did a lot of things overnight.
We computed the, the, um, you know, the, the, uh, recommendations for the next day overnight and in the long run, there wasn't enough, essentially there wasn't enough night to, to do all that. So we though that we had to do it in a different way, but it started out with just doing offs offline and just creating those databases.
Dave Schappell: [00:23:18] So that's an example of how you sorta had to build it and scale it and rebuild it on the fly. Right. Because you built version one, we were like, oh, we can do this overnight. And that works in a couple of weeks or months. And like, always like, we can't do it overnight. So now we have to figure out a way to do it more continuously or yeah.
Eric Benson: [00:23:34] Even in the beginning. Um, I think, you know, It started out even at the, at the launch, it was, there was a weekly build in a daily, overnight build. So there were some, you know, and over time that could have I'm sure that would just got to a point where it had to be continuous. Right? Yeah. 
Dave Schappell: [00:23:55] So I don't want to spend the whole time asking about the recommendations.
It's just that everybody can relate. No, but everybody can relate to it. But, uh, a few weeks ago I interviewed Josh Peterson and Matt brown, who sorta worked, I think a couple of years later after this launch. And along the way, I think Greg Linden, who was another engineer, I think he bridged it with something that was called instant recommendations.
Was how was that different technologically or was it. Getting new placements, like on the shopping, in the shopping cart, you know, saying, Hey, that you have those two things in your cart, maybe you want to add this third thing or, or was it actually technologically different what he was doing with instant Rex?
Eric Benson: [00:24:33] Because I think it was, um, just different ways of exploiting the same data. So with the data that we had built that similarity, so it's associating items to other items could be then used to generate recommendations in other contexts, like you could see recommendations based on what's in your shopping cart or recommendations.
Dave Schappell: [00:24:53] You haven't necessarily bought those items yet, but those items or an item that you're looking at, right. You could say. 
Eric Benson: [00:24:59] Or you could get recommendations based on your previous purchases. You, you know, like, oh, um, you know, like you could just, just on the, and I think we ended up doing this later on the front page.
When you, when you come to the store, it would say, oh, based on your purchases, we think you would like this, this item. 
Dave Schappell: [00:25:15] So yeah. I love how good it works when I've left something in my cart and they recommend it to me again. I'm like, how did they know? Right. As I left, I left it sitting in the shopping cart.
So, so one last question on this then is maybe you didn't have to worry about it with V1, but how did you, uh, how did you reduce the impact of like new releases or really popular items like Harry Potter? You know, which, which, again, that was later, but, you know, how did, like, how would you promote unusual items so that they showed up sometime?
And it wasn't just super popular items. Like, do you, do you remember any of that or is that I remember? 
Eric Benson: [00:25:52] And I think that that was an issue that we, we knew about and we started working on even in the very first version and it was, um, let's see if I remember. Yeah. So we sort of, we were 
expecting that we were expecting 
like, oh, well, aren't we just going to see the most popular items overall recommended.
And so I think there was. Some bias given to so basically to try and filter out the most popular items. But, I think what we realized was, the very popular items would have other popular items recommended or associated with them. And the less popular items would have less popular items show up because of each other 
Dave Schappell: [00:26:45] Yes so if you went to a really niche item then the people that would want to buy that niche item also would buy a whole bunch of other niche items that would show up a lot. more 
Eric Benson: [00:26:52] Exactly. Because they're buying based on their interests, that might be not mainstream for whatever reason. 
Dave Schappell: [00:26:59] That's interesting. Yeah. So jumping to another topic, which is web labs. So we talked about this during episode 10, with Josh Peterson and Matt  did. And I think you told me, did you build V1 of a level and maybe if he's built V1, do you know how many versions you built or did you hand it off immediately?
Eric Benson: [00:27:18] All I did was build the initial version, which was extremely primitive and, and I, I honestly don't even know what happened next with that. Um, but it was, uh, essentially just, um, and I, I don't know how much you want to cover this material, but it was a way to, it was also also called AB testing where you could have present different content to different users.
So one set of users would see one version of the website and a different, and another set would see a different version of the website. And then you would compare their behavior. Right. Um, so it was basically doing, running an experiment where you, you know, look to see how that, how those changes changed, changed the way that they, that they behaved.
Dave Schappell: [00:28:05] And like, was that a unique idea or is that something that, that you had worked on at a different company? Did it come from Juno? Did it come from Jeff? I, you know. 
Eric Benson: [00:28:15] It was something that was in the air. And I think that, um, it was something that had always been talked about, um, not just for Amazon, but, uh, it's something that websites could do, you know?
And I, and I don't think it was, uh, it certainly wasn't unique to Amazon and I don't, I don't know if anyone else had done it previously, but it wasn't a, it wasn't a novel idea. It was just, uh, you know, something that we knew because the nature of, of the internet was such that we could do something like that, where, because we created a session.
ID for each user. And so each user, when they came in would, would, uh, be identified and then their pages would be identified with that session. Then we could, we could separate some sessions into one group and then other sessions into another group. 
Dave Schappell: [00:29:07] Yeah. And what's interesting. I mean, it's very similar to what we're talking about with book recommendations.
It's not something that a physical store can do, you know, you walk in and that store is the same store for everybody. Whereas, you know, you're interested in computer books and I'm interested in, you know, I'm a parent who has interested in buying children's books, where it should be a totally different experience.
And that's what that, that's an extreme version of long-term personalization, but this is just like, What a red button worked better than a green button or, you know, putting these things in different placements work better. Um, by the way, I did find out at least one story is that Peter Morris, who was a data, I think analytics engineer, I think he, uh, I think was this tightly it manager, web analytics.
He said that he thinks the first AB test was testing the color of the add to cart button on books, detail. And it had, and that it had little to no impact. Uh, but he assured me that many future tests had massive, uh, impacts. Uh, so, um, which I thought was interesting. And just back at Amazon again. Yeah. So what made building web, just not going into excruciating detail, but what made it really difficult?
Because I I'm, I'm not trying to trivialize it, but like for somebody listening and they say, well, they come in, you assign. Session IDs. They get each, each one gets a different treatment. And then you look at all the, the, what happened at the end, which one bought more, which one spent more time on the website, like just massively difficult.
Eric Benson: [00:30:31] Yeah. It's, uh, it's simple in concept it's just difficult and execution for, um, and the main, well, a lot of reasons, but, um, because we, we had our systems in place for doing things like content creation and content management and, um, site management and, um, and collecting logs and analyzing logs. All those things were, um, hand-built they were built in-house and, um, you know, on the fly and not, they weren't really.
They, they weren't high priority items that was sort of infrastructure that, um, was lower on list. So that the things that were high on the list were building new businesses. And the things like, you know, the basic structure of things was low on the list of priorities in the company. Right. Um, so we had things that were essentially jury rigged.
Um, you know, one-off, you know, things that, that we had built in-house and that worked, but were somewhat fragile, not design, not, uh, not flexible, not designed to, to be modified easily. And so going through and, and changing every one of these elements to allow for multiple treatments and multiple sessions, different groups of sessions and so on, uh, was challenging.
And then even at the end of it, having being able to take the results and do any analysis. Uh, also challenging. Right. 
Dave Schappell: [00:32:16] And was it, was it so web labs and AB tests, but were the, was the terminology always like very, uh, was it chosen for a reason, like, because this is how experiments are run. 
Eric Benson: [00:32:28] Yeah. So I, I tried hard to, um, use the terminology of scientific experiments to, to do it in a way that, so that when, uh, we had, you know, actual scientists designing experiments that they would, they would look at it and be, it would be familiar to them.
You know, the, the, the concept of, um, creating an experiment where you go from, uh, uh, treat treatments, different treatments, and then dividing the users into treatment groups and then, you know, looking at the results. So, um, yeah, so I just, it was. Yeah. So just using that terminology of call it, calling it the web lab.
So it's a laboratory, you know, and you're doing experiments. And so that, that concept was, you know, and it, it was, you know, sort of colloquially called AB testing at the time. But, but I think I wanted to, to use that more scientific terminology. 
Dave Schappell: [00:33:22] So, well, well, Peter Morris made the point that, who, however, it was decided, but then AB testing was like, he said it confused some people because they thought that you could only do a version a and a version, Brian.
And he sent me a funny picture that I'll include in the post, which was, uh, they had in their office, which had a switch for a and B. Um, but really you could run 10 versions of a test or, you know, it's not limited to one. 
Eric Benson: [00:33:45] Yeah. And I guess that's another thing about scientific experiments is that you generally have a control group and then treatment groups and in our world, in some cases there would be a control group in the sense that it was that there wasn't any change.
Compared to what it was before, but it doesn't exactly make sense to say there's a control group because it's like, it was constantly changing anyway. So just changing it one way and then trying to get another way. Yeah. Anyway. 
Dave Schappell: [00:34:14] And customers didn't notice, I mean, there's two questions here. Customers didn't notice it then.
And I mean, right now, probably Amazon is running hundreds of web labs, we would assume. Right. They're probably testing things everywhere. Right. Um, but there was no, there was no impact. W when my mom's shopping on the site, she wouldn't notice any difference. It's just that behind the scenes, my aunt might be looking at an entirely different version of that webpage and you're aggregating all their results to, to see which version converts.
Eric Benson: [00:34:42] Sure. If you were, if you were an active shopper at Amazon and came to the site multiple times a week, say you might notice the site was changing, but you wouldn't really know why it was changing. And you wouldn't know that you were getting something different than someone else who was using the same site.
Dave Schappell: [00:35:00] So, yeah. And now it's 24 years later. So when you think of the compounding effects of, you know, probably hundreds of thousands of web labs over the years, and hopefully you're not going too far in reverse on any of them, that's sort of where the advantages build up. And it becomes difficult for other websites to copy because they don't know which web labs are working or effective and which are. 
Eric Benson: [00:35:21] Well, they can do their own if they have, if they have that technology, there's nothing magic about the technology we were using.
I mean, in fact it was, it was very primitive and I don't know what state it's in now, but, um, it, uh, but yeah, anyone can do it. There was nothing, nothing that difficult, inherently difficult about it, but, um, then using what you do with it, once you have that is something else. And I also think there's a, there is an issue that you can, expecting that we were expecting like if you do too many things that optimize, you know, there, and I guess that there's sort of the concept of, of reaching, uh, a local maximum, you know, where there's a, like a global, like a global increase that you would miss out on because you've reached a local maximum.
And because you're making many small changes to optimize people's behavior, but then not realizing that in optimizing their, their short term behavior, you may be alienating them in the long run that's, you know? 
Dave Schappell: [00:36:23] Yeah. So like something like, you could be really optimizing the book experience, but you're not introducing them to the music and video categories.
And so it limits the overall or longterm. Um, yeah. 
Eric Benson: [00:36:33] Yeah. I was thinking more in the problem of just, uh, you know, if you just aggressively market, if you aggressively sell things on the site, if you just to, you know, like you say, oh, well, if I put. Three images on the site, then it's better than if I put two images and out now, gee, if I put five images on the site and then it's even better.
And then ultimately like what happens is that people just get burned out because it's just, there's too much clutter and you don't realize it because you're, you're T you say, oh, but if I put another image, then people like it better. So people pay more people spend more because they're there more things to click on, you know, that sort of thing.
So, yeah. Yeah. 
Dave Schappell: [00:37:14] So in, in, I think it's related so early on all of the detail pages, a product page was kind of hard coded, right? And in order to run an AB test, you kind of have to be able to dynamically serve. One version of the page and the other was yeah. Was cat sups used for that? And if so, can you maybe give some background about what cat subs was and then sure.
I think Paul Davis built it. He was a engineer, number two, you know, mate, and I think that's related to web labs, but tell me if it is. 
Eric Benson: [00:37:50] One element that you could use in creating different content for different users, but cat's subs was, uh, just a, a template system. And if you're working in, uh, building HTML pages, this is going back to the nineties.
Um, basically ACML pages were static, just, you know, code that would, um, generate the content on a page. Uh, but, um, pages like, like detailed pages where, uh, the, the product pages were, um, dynamically generated from a database from, uh, from a. Customer from an item database, basically. Right. And, um, so, uh, there, we needed to be able to, um, make dynamic substitutions into those, into those pages.
And so cat subs was basically just a way of putting, putting marks in this, in the HTML files that would notify the, the, uh, software that serves it on the site that they had to make substitutions based on looking up something in a database, I guess that was another project that you had, which was to improve, uh, ups.
Yes. So, um, yes, uh, Paul Davis had built, uh, a simple version of this for the initial site. And, uh, it was kind of slow and kind of limited in and had a very, very small set of commands you could use to, to, um, make substitutions on the page. Uh, and, and at the time that we launched music, we started to need more capability.
And, I think it had started before that, but it became an imperative It, turned into a project that got assigned resources. I should say. 
Dave Schappell: [00:39:42] That's an example of a command. Well, what's an example of one or two commands that so people can understand what you mean, like what it was able to do versus what it needed to be able to do as you added new, new cat.
Eric Benson: [00:39:53] Um, well, so typical thing, while even if you look at something like similarities, so if you are looking at a detail page, you'll see say an image of a product and you'll see the name of the product and, you'll see the price and so on. and all those things are looked up in a database and there would be commands to, show those, values on the page. So that, that there were, uh, basically markers in the cat subs language that would say, you know, show the price or show the, you know, so on. And, and then when we added similarities, we had to do the same thing with similarities so more and more, there were things that had to be computed, and built, on the fly.
And so, uh, I guess, um, we, we experiment experimented with things like, oh, uh, we wanted to show, we were deciding what, whether three showing three similar items or five similar items would be more useful. And, uh, so we, you know, anyway, we, we needed to be able to do, um, you know, change those easily without having to write a lot of code.
Dave Schappell: [00:41:09] And I would assume all of these things are slowing every time you're adding some fancy new recommendation or similarity or just content overall, it's slowing these pages down. Is that fair to say or no? 
Eric Benson: [00:41:21] Yes. Uh, there's more work you have to do then for every page. Yeah, absolutely. So, you know, the simplest HTML page, uh, it just, it's just copying from a file on the, on the web server.
It's just copying those out to the, the, the network. Um, and then everything that you do, that's different on the page, like, you know, for the customer or for the item has involves database lookups or other other work on the, on the server side before you can then generate that data to send out to the network.
Dave Schappell: [00:42:00] And in hindsight, I mean, Your modifications were the right ones at the time. Do you think it should have been done differently? It's not really criticizing anyone looking back on it now. And you're like, boy, we really would've saved a lot of pain. Had we done something a little differently earlier on, 
Eric Benson: [00:42:20] I think many things were done early on with a sense of expediency like that.
They're just, you know, what we have to do to get things working right now. And we don't have a lot of time to think about it. And, uh, just to, you know, and cat subs was definitely one of those things that we knew was not the right solution, but it was a solution that worked at the time. And, um, you know, it wasn't, if I think that the problem with cat subs, I mean, there were performance issues that wasn't the biggest problem though.
The biggest problem with cat subs was essentially it was creating a very sort of primitive programming language. Uh, for people to re to create those pages to great those templates. And, um, there, there was always a separation that, you know, there was code that professional programmers road that was running on the system and written in a language like C or C plus plus at that time and other languages later.
But at that time it was CNC plus. And, um, then there were, um, things like the pages, like the HTML, which was developed by the company tent team, which was not professional programmers. There were people who were, whose job was to lay out layout elements on the page. And, uh, you know, so there were more in the area of design than in programming.
Um, and we had to give them a way to. Make those customizations, but, uh, that we were deliberately trying to make it not be a programming language. Right. But ultimately you need the power of a programming language to get that job done. And so it ended up being a very poor programming language essentially. 
Dave Schappell: [00:44:16] Which will live for a very, very long time.
Eric Benson: [00:44:19] Yeah. Uh, yes. And became, and it, because it was not a, uh, you know, an established programming language that people could learn about outside of Amazon. It became sort of a dark art that people learn when they joined the content creation team. 
Dave Schappell: [00:44:34] Yeah. You couldn't just go hire experts or people that worked on something that was very similar things. It was a, yeah. 
Eric Benson: [00:44:40] One of the jokes that went around was, uh, somebody had had mocked up an O'Reilly book. Uh, about cat subs, you know, that the cover of the book, you know, just as a joke and it's like, yeah, no, there is no O'Reilly, that's kind of what was needed. Exactly. 
Dave Schappell: [00:44:59] So, yeah. So making, making a big jump, uh, we haven't talked about the switch from the company from Unix to Linux as a gigantic change, right?
Could you maybe just give a short background on, you know, what things were and that no one's talked about it yet. I just, uh, found it interesting and I traded some mails with people and said, Eric would be the, one of the good people to talk to about that. So maybe just explain it to people, why it was really important and how it came to be.
Eric Benson: [00:45:31] And, you know, you have to kind of look back to the history of the internet and, um, you know, hardware systems at that time, it was in the mid nineties at the, when, when shell started development. Amazon version one in 1994, uh, it was on sun workstations, which was a sensible choice at the time. Um, and sun had their own version of Unix.
Um, and so the software was built using, um, using that version of Unix was Sonos at the time. And, um, but then I think really just, just before I started, so not long after the launch, um, it was moved from, from Sonos to digital equipment corporation, digital Unix, uh, which is, uh, another, another company that no longer exists.
But, um, but yeah, so, uh, at the time the digital Unix systems were, uh, they were built on a 64 good process. So they had the opportunity to, to, to scale up. We thought maybe would scale up better. Um, and we use the digital units and, and its successors, uh, from, from that time until the year 2000. And, uh, the problem was that by that time we had, um, gone from one server, one piece of hardware to, um, I guess by that time it must've been thousands, I guess I don't really know the exact numbers, but you know, it went from, from one to three to dozens, to hundreds, to, you know, so at any rate, there were, there were many, many servers.
Um, so the hardware costs were, were significant. And, uh, because it was, um, uh, a small manufacturer and there were powerful machines. The cost per unit was very high. So that was, um, and then, uh, we had, so the company knew that we had to do something about that and we had an opportunity to make a major purchase from HP.
And this was early in 2000. And, uh, so in order to take advantage of that, we had to, um, either get the software running on windows, which was, would have been almost impossible to do, uh, either that, or get it running on Linux, which at the time was, was still an unusual choice, but starting to get more popular as a, as a server software.
And right, much later it became sort of a standard for server software, but, but it was still, it was still an unusual choice, but, um, it, so basically we, we decided we needed to port from port our software from digital units. Linux running on inexpensive PC hardware, which is what HP was offering. So, and in the process we would be saving.
Um, and I don't know what the, I don't know the numbers, but it was saving a significant amount on each unit of, of competence. 
Dave Schappell: [00:48:49] Michelle I'll jump in there. Cause I looked today, there was a scene, there was a scene at article and this was in May, 2002. So it was already a year or two after, but they said, um, that they, they reported then a quarter of the change cut technology expenses by about 25% from 71 million to 54 million.
And then just think, we'll think the ripple that through 20 years explosive growth in AWS. So like likely save billions of dollars from that, that painful project or so such. So jumping back to then making it work. Did you, did you get it running locally? Like how did you even figure out that it was possible?
Eric Benson: [00:49:30] It started as. Really, uh, maybe a year earlier. Uh, I had, um, just as a sort of, you know, just a skunkworks kind of thing, like a personal curiosity, I'd taken, just try to try to port the software to run on Linux myself, um, on just on one machine to see if it could possibly work. And, and, uh, I was able to do it and I didn't, I didn't pursue it because it was not considered a priority at the time.
But, um, I basically got this thing running and I said, Hey, look, you know, this works and it's not done, but it's, it works well, it's clearly going to work. If we want to do this it's function, it's functioning. It's not like not functioning. Right. And so, um, when this opportunity came along to, to purchase hardware and it became clear that that running on Linux was the only way to take advantage of it.
Um, It was basically a team of two software engineers. That was me and Bob , who I'd mentioned earlier was, was responsible for the database with when we had that outage. Um, so the two of us pretty much spent the entire year, 2000 on, um, just making sure that our software actually worked on both systems during that entire year.
So everything that, that, that large, by that time, large team of software engineers was doing, had to, had to work on both systems and, uh, because people were, were developing entirely on digital units. They were essentially unaware. They had no way to test that their server was working on Lennox. So I had to go through essentially multiple times a day and just make changes to every piece of code that was checked into the system to make it, to keep it running, to keep it working on Linux during that entire year.
So, um, so it was sort of a constant, it was like running on a treadmill, uh, to, to just keep things going through that year. And, and in the meantime, Bob was working on, on all the pieces surrounding it, having to do with, um, Deployment. So there were a lot of scripts and, you know, just pieces of code that, that, uh, were involved in, in just deployment, basically getting, getting software from, from the development servers to test servers, to the production servers and so on and collecting logs and all that.
Just things that, you know, kept the site running, not the, not the core software. So I was basically working on the core software and Bob was working on all the other pieces of the system and. 
Dave Schappell: [00:52:13] We just turn it over, like flip it all over, did it, wasn't like we'll roll a percentage of our traffic and see how it works or did it work out at the gate.
Eric Benson: [00:52:21] So, um, at the end of that year, uh, we started to get some hardware in and then we, and we started at that point. We, we flipped not the entire system all at once, which, you know, would have been a recipe for disaster. But yeah, we, we, I think it was, and I don't know what the percentages were, but yeah, we started.
Routing some of the, some of the, um, traffic to the Linux servers, like at the point where we thought that the Linux system was working in the same way as the digital unit system, then we, we ran traffic over there and, and it started working. And so it, and it wasn't very long after that, that we completely switched over it.
Wasn't, you know, there really, once you get to that point, there's no point in keeping both of them running. It's actually. 
Dave Schappell: [00:53:10] When you want to run as fast as possible and fix any issues. 
Eric Benson: [00:53:14] The sooner you can decommission the old system. It's great. It's going to be, yeah. 
Dave Schappell: [00:53:19] Yeah. So we're running out of time. I thought we have to talk about Rufus, at least one manager to, uh, uh, so, uh, maybe just quickly tell the story about, uh, Rufus and site launches, which is one of my, you know, and yeah. 
Eric Benson: [00:53:37] Well, so, um, Of roof as well. Um, so Rufus was our dog in San Francisco before we came to Seattle. And, um, when Susan and I just, it was just the two of us and our dog. And, um, we were, when we started at Amazon, it was, uh, we knew it was going to be long days at the office. And we were working in a, in a building that was not very fancy, in fact, pretty, pretty low key.
Um, and so, uh, we just talked to talk to Jeff about, um, you know, it would be okay to bring the dog to work. That was sort of the, the essence of it. Is it okay if we bring Rufus to work? And, and, um, and Jeff said, is he a good dog? Which was a funny, funny thing to say, but, uh, and, and we said, yeah, he's a good dog.
And generally he wasn't good, but yeah. So Rufus came to the office nearly every day. For the time that we worked there. Um, and, uh, he kind of became the company mascot and then he went, he would wander from office to office and go to meetings and, you know, just hang out and yeah, by her did, uh. 
Dave Schappell: [00:54:54] Didn't you have to get written into one of the leases for one of the buildings as an exception, because later dogs are everywhere, but I guess early on, they were in a building where it wasn't allowed.
Eric Benson: [00:55:03] Yeah. Um, so we moved from the, uh, Soto office to, uh, the, the Columbia building downtown. And that was also not a very fancy building, but, uh, the, the, uh, the landlords really didn't want to have any pets in there. 
Dave Schappell: [00:55:23] That building was awful. They had no right band from that though. 
Eric Benson: [00:55:27] Um, but yeah, but they really didn't want to have pets in there and they're, you know, fears of liability if someone got hurt and then, you know, those, those things.
So, um, The landlords asked Amazon to sign a waiver for, to hold them harmless, if anything happened because of the dog. Now, the funny thing is then Jeff asked us as Susan and I to waiver. So anyway, so, you know. 
Dave Schappell: [00:55:55] Did you actually sign that? 
Eric Benson: [00:55:56] Yeah, sure. It's like Rufus groups didn't, you know, pretty confident.
Dave Schappell: [00:56:01] And then the ones, one other story is that Rufus used to, uh, push the button to launch versions of the website. How many, how often did that? Is this the mascot? 
Eric Benson: [00:56:10] So everybody loved Rufus and, you know, switching the site over became a big thing that we did late at night and, you know, we would usually go out to dinner afterwards and things like that.
So it was, it was kind of a big deal when that happened. And I think might've been marrying Mohit who. I think you probably have talked to, and I think she might be the one who said, Hey, let's get refers to, to switch the site, you know, and stuff. And of course, Rufus was just there. So, you know, that just basically meant picking up his paw and having it push, push the button on the keyboard.
Dave Schappell: [00:56:48] I was there for a few of those. I was there for the music, one video, one and auctions. And so that's pretty vivid memory and we have a photo of one of those in, I think Merriam uploaded it. So I'll, I'll copy it over on your episode as well. And there's a building named after Rufus down in south lake union, right?
Eric Benson: [00:57:06] Absolutely. Yeah. Yeah. 
Dave Schappell: [00:57:08] You should demand payment for that show. Oh, your contract. 
Eric Benson: [00:57:12] And the entire, like, there was a whole building project that was called Rufus two dot O. Right. Yeah. 
Dave Schappell: [00:57:19] Awesome. So, uh, JJ Jacoby, who we mentioned early, earlier, Amazon engineer, you worked with before. So she'd said about you a standout area that resonates in my memory about Eric was how he mentored engineers.
There were so little time to do that. He always made time for newer engineers that wanted to discuss systems, languages, projects they were designing. And she just wondered if that stood out for you, because she said it helped people build up during a time of heavy growth. And then I'll add probably at a time we had very little formal training.
Is that something you agree? 
Eric Benson: [00:57:51] Yeah, absolutely. That was something that, that I, you know, I think I, yeah, exactly gravitated towards and, and to the point where I think for, for most of the, like the, say the, the third and fourth years that I was there, that was essentially my job. There was, I didn't really have any other projects that I w it wasn't until my fifth year that, that we did the Lenox project and that was.
Heads down really didn't do anything else, but really in my third and fourth years, what I was doing was I was just a resource, basically a source of knowledge about, about how systems worked in the company. Right. And, uh, you know, so when, whenever someone was launching an new part of the site or, you know, on a different platform or for whatever reason, you know, they were doing something new, right.
They would come to me and say, well, how would you do this? And I'd sort of like, help them think through. So it did a lot of brainstorming on how to, how to, you know, add new features to the site and so on. 
Dave Schappell: [00:58:52] And part of that was probably just because you had touched every part of the site as it built from the early version that shell and Paul built to something not gets increasingly complicated, but these new people were coming in with none of that background. Is that right? 
Eric Benson: [00:59:06] It's right. And then just, you know, right. So I just been there, I just kind of built so many things. Yeah. You know, so many parts of the system over time. I mean, I think the, that the web lab was probably a good example of that or because it touched every, every phase of the system and, you know, but yeah, so there were just a lot of different pieces and there was a lot of, um, you know, sort of arcane knowledge, I guess you could say that, you know, people needed to get their job done and, and, you know, there wasn't any way to just know about it. Dude. Someone had to tell you. And so, yeah. 
Dave Schappell: [00:59:44] So, so I don't think that term existed. You mentioned about, we accumulated a lot of technical debt over the years, but that's because we had to build it all like that Oracle database and was what shell and Paul built in that first sprint. But then you were sort of building while you were scaling while you were fixing, you know, like.
I mean, it's nice to say, should have done it differently or more elegantly, but was there a different way to do it really given how fast just everything was grown? 
Eric Benson: [01:00:13] Um, yeah, I think that, um, speed was the first thing that everybody wanted, you know, speed and growth, so that things had happened fast and they, you know, we had to just deal with whatever came our way.
Uh, and, and we didn't have any products we could buy to solve those problems. We had to, we had to build everything. So, I mean, most companies go through this build or buy decision, you know, oh, do I, you know, am I going to spend money on, uh, you know, some it system, or am I going to just fly people in and build something in-house that will do it.
Well, you know, no, you can't. Yeah. Yeah. There's no, there wasn't, there weren't any products to purchase to scale websites at the time. So we just. 
Dave Schappell: [01:00:57] And the one you did buy, I didn't work for the, for the city. You know, the one we started the interview talking about the, not that they, that there was anything wrong with what they did, but you know, probably didn't work at scale didn't work because of the user interface didn't work for a lot of reasons. And so, yeah. Um.
Eric Benson: [01:01:12] So I was thinking later, um, in the, when I was, um, during the year, when I was working on Linux, it was almost like Amazon was getting pretty large by that time. And, uh, I started, I started getting calls from people I had worked with in the bay area. And they were now at companies selling products to do website building websites or scaling websites or things like that.
And they were, they were. Trying to say, oh, Amazon should buy this product. And every single one of them I looked at and went, this is a toy compared to what we're doing. Yeah. So like, yeah, your product might work for somebody who's doing something tiny, but it's, it just it's useless to us. So even, even at the point where their work products like that, they wouldn't have been applicable to us.
Dave Schappell: [01:02:05] So I heard from people that was that in the beginning, it was a little bit difficult to recruit engineers. Um, and this is going a little bit off our plan topic, but it just made me think of it. Like they, they said it was a little bit, maybe it was Joel or Kim. I forget exactly Dwayne and Rubin, I think you mentioned.
And they were like, it was a little hard to recruit engineers. Cause in the early days, it wasn't seen as there was a lot of, you know, sexy problems being addressed and from an engineering perspective, where were you there from that? And like, how did that change? Like it did pivot over time where it became a very attractive to me.
Eric Benson: [01:02:38] Um, yeah, I do remember that. And then thinking. I think that, that, you know, sort of essentially like building database applications, you know, the crud applications, you know, uh, which right. Anyway, just like the things that would just, you know, where you're buying a product and, you know, putting an entry in it, a ledger essentially, and, you know, shipping products to people that was considered sort of drudgery, you know, like people wrote COBOL code to do those things.
And so what we were doing was, uh, basically solving old problems, but in new ways and, and it was an a, but it wasn't, it wasn't considered glamorous. I mean, it's somewhat a place like Microsoft was much more interesting to people at that point because, you know, they were building, you know, operating system software and compilers and EV you know, development systems and all those things.
Dave Schappell: [01:03:34] So. Well, that's great. I mean, we, we hit an hour again, without even thinking about it. I actually, I want to like to end on, you know, lessons for entrepreneurs. I mean, in this case, I think a big part. It was, it wasn't about technology. I mean, you just were dealing with, it's just sounds like you're telling these stories and like, I don't know, they actually do seem like sexy projects to me, like the recommendations, um, you know, and, and the, the Linux transition must have been like a heroic thing, but like, when you, when you hear, if you're talking to younger entrepreneurs or even innovators inside companies, what do you think the lesson is, was like, how did you guys stay focused in the midst of all the chaos?
Like it wasn't just make customers happy. Like, was it, was there something about the Amazon mission that really appealed to you and the people that you admired at the company? 
Eric Benson: [01:04:23] Uh, I think, well, once I think it was easy to get sort of caught up in the, in the, you know, Uh, the excitement of the moment. I think we, we did have a sense that we were changing the world and that the hyper growth that was happening, it was hard to deal with, but it was very exciting too, because, you know, we were going from a tiny company to a huge company.
And, you know, I mean, I, I could not have imagined what would happen today. You know, that's like it's gone to a whole different level, but anyway, um, it's uh, but yeah, it was, it was very exciting once. I mean, I, I, I often say, well, I moved from San Francisco to Seattle and joined a cult. You know, it was sort of like, it had that feeling where I didn't really have much awareness of the city I lived in.
I was just going to this place every day and we were just immersed in it. And, and, uh, you know, there were just so many, so many problems to solve every day. We didn't, you know, Yeah. Yeah. I mean, that's been really, I mean. 
Dave Schappell: [01:05:29] I got there in 98, so after everything was already pretty popular, you know, like lots of news were good and bad, but it must've been pretty exciting though, getting there in 96.
And then when that New York times article hit in 97, I think, and you know, you started getting like, just like, oh my God, this is just take, you know, everything you're planning for your like you'll plan for four X growth. But two months later, you're at four X growth and you have to rethink, or it must've been so good.
Eric Benson: [01:05:54] Like there was a, there was a 60 minutes story about Amazon. A lot of this stuff happened in 99 and 2000. There was a 60 minute story. I remember, uh, Jeff was on the tonight show with Jay Leno. He was a man of the year in 99, I think. And on the time magazine man of the year. So there were all these things that we'd gone from virtually, nothing like this tiny, tiny business in this little, little shop.
And anyway, so. Yeah. And then it was like, wow, what's happened. You know, like it turned around and like, yeah. 
Dave Schappell: [01:06:31] That's back when the interviews were all very popular and positive and they remember like Barbara Walters and, you know, interviewing Jeff and stuff. And I also always tell the story. I like, cause cause that second avenue building was so small, you'd like be walking down the hallway and like Stephen King would be coming out of your wife's office.
Like, you know what I mean was like, it was, it was really fun just because every, it was just so compressed, you know, doing interviews in the stairwells, like as bad as that is. It's still also, uh, good memories as well. It's a very compressed yeah. 
Eric Benson: [01:07:04] Yeah. It was just caught up in the whirlwind and yeah. Yeah.
Dave Schappell: [01:07:09] Well look, thank you so much for being my guest on the podcast. Um, you know, I think people are going to love hearing these stories. I mean, literally. I think your names come up on every, almost every episode, especially, you know, with, and even with the pre episode chats with people like Greg Linden and others.
So I was glad to be able to get some more details on, on those and, and, you know, it was also just great to see you. Uh, it's been a long time since we've been in Seattle. 
Eric Benson: [01:07:34] Okay. Well, it was great talking to you. 
Dave Schappell: [01:07:37] Yeah, you too. And for the audience, thank you for listening to the event like an owner podcast. If you'd like more details about what we discussed or want to contact me with edits, please visit invent like an owner.com to sign up for the weekly newsletter.
And that's it for today. Remember? No sniffling. .