Jeff Bezos (Amazon.com) - "Earth's Most Customer-Centric Company: Differentiating With Technology”

[MUSIC PLAYING]

HOST: So I don't think Jeff Bezos needs much of an introduction. You've perhaps heard of this small company he runs out of some city in the west coast--

[LAUGHTER]

--where there's another company. I took a poll of the faculty recently and-- I was at a faculty lunch and we were talking about what people did online. And I sort of ask people questions and eventually I said, well, how many of you have bought something on amazon.com. And it was actually 100%, I think, raised their hand.

BEZOS: Yeah, baby!

[LAUGHTER]

HOST: I guarantee it was not one of those.

[LAUGHTER]

Any rate, Jeff is a graduate of Princeton. And I'm pleased to say, as he was telling me upstairs, he graduated from Princeton when it was still an EECS-- an electrical engineering and computer science department. Before the department bifurcated. And so he does have an electrical engineering and computer science degree. And I think-- he tells me he's glad he has one of those. And as he should be.

After leaving Princeton, he went to Bankers Trust. He worked there in the development of computer systems. So presumably if you need help with code, he can help you with some of your programming and stuff of that nature.

After that he went to D.E. Shaw. Well known for their high-tech approach to finance. And in fact, one of two financial services companies-- three, I guess-- that belongs to our industrial program in the department. And then he left D.E. Shaw to start amazon.com. And I'm sure he can tell you more about that than I can. Jeff.

BEZOS: Thank you.

[APPLAUSE]

Thank you, you guys it's great to be here. So I'll start out telling you a little bit about the founding of amazon.com and then I'm going to do my favorite thing which is to give a live website demo.

This is the house where Amazon was founded. On the left-hand side here is this little enclosed garage. I was living in New York City, working for D.E. Shaw and Co and I came across the fact that web usage was growing at 2,300% a year. This was one of those weird things where you could measure the rate of growth without actually knowing the baseline usage.

And people were doing that because they could put sniffers at various points-- at various nodes-- and they could sort of see how fast the traffic levels were growing past those nodes. And they could get a statistical sample of those but they actually had no idea what baseline usage was.

But you could look around the web already, in the spring of 1994, and see that it was big enough that if it was growing at 2,300% a year, pretty soon it was going to be huge. And so I kind of packed up my wife and I and we flew to Fort Worth, Texas where my dad gave us a car-- a 1988 Chevy Blazer. Which by the way, Consumer Reports recommends not to buy used under any circumstances for any price. And we drove that car from Texas to Seattle.

Seattle was chosen as the best place to start amazon.com because it was a large pool of technical talent and also nearby at the time, was the largest book warehouse in the world. In a town called Roseburg, Oregon.

So we got there and the first order of business was to recruit a V.P. of Engineering. I talked to several people and I found the guy I really liked, his name was Shell. He was with the company for six years and was a huge instrumental force in the company and in the early days-- not single-handedly of course, but in large part-- led all of the software engineering that was amazon.com.

He had nothing to do with the look of this website, though. This is what our website looked like when we launched in 1995. I wrote all of that HTML myself--

[LAUGHTER]

[APPLAUSE]

--please hold your applause.

[LAUGHTER]

But Shell did the hard part, he wrote all of the code that actually made it work. Before hiring Shell, I was making our desks out of doors and four by fours.

It took me three months to convince Shell to come join this company because it was really just a one person company. It was me and it was a piece of paper and a name. And the name wasn't even Amazon that time, it was Cadabra. Cadabra got changed because whenever I would say it to anybody over the phone, they would think I said cadaver--

[LAUGHTER]

--and I realized that wasn't going to work very well. Amazon was incorporated, by the way, by I-- I only knew two people in Seattle so we're driving out there, I wanted to be moving fast. I wanted the company be incorporated by the time we arrived.

So I called one of two friends I had in Seattle and said, could you recommend a lawyer who could help me get a company Incorporated and open bank accounts for the company. And do some stuff so I can hit the ground running. My friend said, sure. So amazon.com was incorporated by my friend's divorce lawyer--

[LAUGHTER]

--not like some big high-powered attorney.

But anyway, I finally talked Shell into coming and joining this intrepid venture. But before he had moved from the Bay Area up to Seattle-- he was getting ready to do that move. I called him and I said, Shell, how tall do you want your desk to be? Because I'm making them out of doors I bought at Home Depot and four by fours. There was like 10 seconds of silence on the line and I was thinking to myself, oh great, now he realizes what a two-bit operation this really is and he's going to change his mind. He eventually answered the question.

About a year later I said, Shell, do you remember when I asked you how tall you wanted your desk to be a year ago? You were quiet for like 10 seconds, were you like reconsidering your decision? And he said, no I was just considering how tall I wanted my desk to be.

[LAUGHTER]

So we got the software put together-- took about a year to get the software put together and to write this astonishingly beautiful HTML. It's really ugly, isn't it?

[LAUGHTER]

It is. And if you'll see-- I think the next slide coming up-- no, well this is our next real office. This is what-- I don't have what our website looks like today, but you guys know what our website looks like today.

You will note, that it has some minor improvements like a search box on the main page. Here you had to actually click through to search. You had to click on that link that says, 1 million titles. So things have gotten better.

By the way, since people don't know this, our number one-- the number one thing we've invested in at amazon.com has been technology. And in fact, since 1997, we have-- in the last five years-- we've spent $800 million on technology. So when you look at what it is today and look back at this-- that's what $800 million will buy.

We've only spent $300 million on five million square feet of fulfillment centers, for example. And only $600 million on marketing. So technology has been our single biggest investment and the company basically runs on computer science. It's either that or it's just my bias since that's my background. But at any rate, that's what we do. We really focus on trying to differentiate our experience with the engineering and the algorithms that we put into it.

So anyway, we got this company just about ready to launch and we looked at it and looking at this little tiny fulfillment center-- we were here by then, in this building. We had a basement fulfillment center. To call it a fulfillment center was very grand. There's a lot of puffery. It was 400 square feet, which is about the size of a one car garage.

We launched this business and we're looking at it and we didn't know if anybody would order from us. We really didn't. And in fact, one of the software engineers looked at this little space and he said, I can't figure out whether this is incredibly optimistic or hopelessly pathetic.

[LAUGHTER]

Indeed, we didn't know. We had no idea whether anybody would want to buy things in this way. The business plan called for generating sales very slowly as customers change their attitudes. The original business plan-- which I thought was very optimistic at the time-- called for amazon.com to generate $70 million in sales in the year 2001. We actually generated in excess of $3 billion in sales in the year 2001.

We knew, by the way, that we were really on to something in those first 30 days. In the first 30 days we got orders from all 50 states and 45 different countries. With not a dollar of advertising, just all word of mouth. In the first year we didn't spend anything on advertising. All of it was word of mouth. And that really forged the company. We were going to focus on customer experience. Because we saw the power of word of mouth so very, very clearly in those early days.

Something great about word of mouth online, by the way-- and feedback from customers-- which is that email turns off the politeness gene in the human being. It's wonderful. So people actually tell you what they really think. Usually in all caps.

[LAUGHTER]

So we get lots of info and feedback from our customers and we try to use it well.

Well, in that first few months, we got one order-- it was literally in the first six months, I remember exactly when it was-- an order from Bulgaria. I didn't even know they had internet access in 1995 in Bulgaria, but they did. This person did not pay with a credit card. They paid with cash. Which is a method of payment we discourage.

They had taken two crisp $100 bills and folded them up in a tiny little package. And they took those $100 bills and put them inside a floppy disk. They opened the little metal door and slipped them in the floppy disk. And then they mailed us the floppy disk.

And on the note-- on the floppy disk was a little note and it said, the money is inside the floppy disk.

[LAUGHTER]

And then it went on to say, the customs inspectors steal the money, but they don't read English. So we opened up the little door sure enough, there was $200 in there. They had written their order number on it so we could associate it with-- and we shipped them their books and all was well.

Our vision has changed a little bit but it hasn't changed in about four years. About four years ago, we came up with the notion that we wanted to be a place-- build a place where people can come to find and discover anything that they might want to buy online. Literally universal selection. Anything with a capital "A."

We had already been on the path of trying to be what we call, Earth's most customer centric company. And we mean three things. We have a very precise definition for what customer centricity is at amazon.com. It means listen, invent, and personalize.

The basic idea is you can't really run a business of any kind, I think in most people's opinion, if you're not going to listen to your customers. That sort of fundamental. But invention is equally important. And invention, it goes far beyond listening to customers. Because oftentimes, customers don't really know what it is that they want. It's your job to invent on their behalf and we've done a tremendous amount of invention and innovation of different kinds in the company. I'll tell you a little bit more about that when I give you the live demo.

And then finally personalization. The notion of customer centricity is to build a place where each individual customer has his or her own website. And to do that in large part, not by explicit information that the customer provides, but through implicit observation of their behavior on the website. So that we personalize that experience based on the actions that they take.

And we put a lot of effort into that. We've been working on that for-- well for all but for the first year of the company's operations. So for, I guess about six years now. A lot of-- some of our smartest people have worked on that. There are bunch of areas where good algorithms, good heuristics, and so on, turn out to be very important for driving our customer experience.

And that kind of discovery-- accelerating discovery for customers, which is what personalization is largely all about, is critical when you use this. When you decide to be selection intensive. So we have over 28 million, or something like that, items in our catalog. We have-- across all categories.

When you have so many items, you have to work hard to build tools that help customers find products. But you also have to work hard at something which is a little less intuitive and actually, technically, even more challenging. Which is to help products find customers. So we reverse the sense of that. We've put a lot of energy into helping products find customers and it's paid off.

This is the basic thing that we work on every day. So what we want to do is have the world's best customer experience. And there are a few things that feed into that. Massive selection-- which requires great discovery tools-- and low prices. Those are the two biggest things.

Low prices-- and both of those things-- the heart of both things is technology. It's very interesting because I often get asked-- and again maybe it's just my bias because I'm an EECS geek-- but how our business differs from traditional retail.

The answer to that question-- I happen to know the guy who does real estate for Starbucks. And believe me, this is like the guy who has the metaphorical corner office. I mean, that business is all about real estate. This is why the old retail saw-- the three most important things in retail are location, location, location. And in our business it's technology, technology, technology.

And one of the great things about technology as opposed to real estate, is that real estate on average-- economists will tell you-- will get more expensive at the rate of inflation. So real estate gets more expensive every year. And that's the core ingredient of physical retailing.

Whereas our core ingredients get to follow Moore's law. So disk space getting twice as cheap every 12 months. Bandwidth getting twice as cheap about at that rate. CPU every 18 months. This is a tremendous advantage.

As those things get cheaper and cheaper and cheaper-- in and of themselves, they don't offer any real benefit to customers. But what we get to do is take all that compute power and layer innovation on top of it to really change the customer experience game. And that's a lot of fun.

We have invested enough in our software that we've become the more-- a big part of what we've done is become more classic e-commerce company. A more classic technology company, too. Where we actually now service-- make available as an application service provider-- is an ASP model, our e-commerce technology-- to others. We also have done-- and I think Rob is going to give you guys a quick demo of our web services stuff in a minute, too. Are we still doing that competition?

FREDERICK: Yes.

BEZOS: Oh, good. This is what-- instead of our 400 square foot fulfillment center, we now have five million square feet of full of fulfillment space. It turns out that operations research and all the logistic stuff we do here is astonishingly similar. When I started learning about these techniques I was like, wait a second this is computer science, they just have different words for everything.

One of the things that's very interesting is, in some of these fulfillment centers, we have two million different items in the fulfillment center. And you can order any two of those and somehow, we have to marry those two items together and get them into a single box and ship them to you. That sortation is very challenging and there are many pieces of it that are challenging-- to do in an efficient way.

All of the most obvious things you would try first turn out-- they work but they don't work efficiently. The cost would be too high. And what you end up doing is, using very sophisticated algorithms to calculate an optimal pick path through the fulfillment center.

you pick a bunch of customer orders at once and a decentralized swarm of people actually pick all these things. And then they have to get married and it's sort of a two stage sort later. But the picking algorithms are like a dynamic traveling salesman problem where the cities occasionally move and disappear. It's like unbelievably--

[LAUGHTER]

--it's actually very fun. And I didn't know-- when I started the business, I thought the front end of the business would be very fun and intellectually stimulating and had envisioned a lot of the kind of personalization features that we could build online. But one of the things that I had not known and not envisioned, is how intellectually stimulating inventory optimization and pick path optimization and these kinds of things would be, too. This is kind of like airline routing and so on. It's a very interesting problem.

AUDIENCE: You wrote your own software for that?

BEZOS: We wrote our own software for that. So it's a great question. In fact, we bought off-the-shelf inventory optimization software. There is no good picking software for the two million items so that we built from the very beginning. We did buy inventory optimization software and it didn't work very well for such a large SKU base.

One of the things that we found, is that our operations are significantly different from most companies for a variety of reasons. Your typical mail order company-- most of these things are made for like warehouse management system and made for mail order companies. And you can buy the things off the shelf but they typically have a different number of SKUs measured in the tens of thousands. And a lot of things change when you scale that up to millions of SKUs instead of tens of thousands of SKUs. And so we had to do a lot of it ourselves.

We bought-- early on we bought some personalization software that also didn't work. And one of the things that's interesting about personalization-- and that's where we put a huge amount of energy and attention-- is that first of all, it has to work at scale with millions of customers. And on peak days we will take orders for and ship 2 million items. And so it's very, very sensitive to doing that-- and we do for unrecognized customers, we do real-time personalization based on their click stream. So there really is nobody doing that.

The other thing is, that we can do active experiments. This is like a really big point that most people miss. But it's one thing to take a huge data set and sort of do backward looking data mining on it. And you can actually find some interesting things that way. But what you can't get from that, is anything that has a feedback loop. Where if you change something, changing that thing is actually going to change the customer behavior.

So one of the most fascinating tools we have at our disposal, is the ability to do active experiments. It's kind of this huge laboratory with literally, 30 million visitors a month coming through. And every time we make a little change on the website, we roll that change out to a randomly selected group of people and hold constant the rest.

So we call it A/B testing. We may do-- half the people see version A and half see version B. They're seeing it simultaneously, so we're holding all the factors constant. And we can actually tell. So if we develop a new personalization algorithm that some smart person inside the company comes up with, and they're really jazzed and they think it's better, we never have to argue about that. We say, well let's try it. And a few hours later, we will know whether it's better. And that is huge because those kinds of arguments are so useless and energy draining.

Come on up. I'm going to introduce-- this is Robert. You're an alum.

FREDERICK: Yes I am.

BEZOS: And Robert has been with us for quite a while. How long have you been?

FREDERICK: About 3 and 1/2 years.

BEZOS: 3 and 1/2 years. We've been to Japan together. We've been through several trials together. All good things. You've worked on our wireless business and now you're doing a lot with web services. Let me turn it over to you.

FREDERICK: So again, my name is Robert Frederick. I manage amazon.com's web services program. How many of you have heard of web services? Or are doing things with web services? Okay, great. Amazon Web Services is something that we launched this year in July. We found that by providing an API-- a solid SDK or an API-- to our broad website developers as well as our associates, that we would be able to get more and more users to interact with our features all over the internet rather than just Amazon's website.

So what we were thinking of doing is, we had three goals. We wanted to define a product. And that product is the ability to create your own store on your own website. To expose Amazon services and features on, I believe it at this point in time, it's 900,000 websites that interact with this on a day by day basis.

So what we were doing is, we were defining a capability to interact. To get search results. To interact with our similarities engine. To find products and to build those storefronts on external websites all over the internet in various countries. All through exposing a solid API and SDK.

So again, we had a market of 900,000 websites. We have developers who wanted to build SOAP packages, Scripts, Perl, PHP, you name it-- .net applications. And we wanted to provide those applications to each one of these web site owners so that they can use our tools and interact with us via soap or via XML over HTTP, in a very easy and viable solution.

So the economic model for this as well-- or why are we doing this? Why would Amazon ever want to expose the services that we have on our site to all of these other websites on the internet? And the reason is very simple. The more people that are interacting with our products, the more items that will potentially sell.

And then we also have the capability to provide these products on their website in a way that doesn't have to make them recreate the wheel. It's easy to implement. Literally, anywhere between five minutes to a couple of hours, and people are able to have a very robust storefront. A very robust way of interacting with our products and our selection.

What we ended up doing was, we came up with a SOAP interface and a XML over HTTP interface. And I just wanted to go through a couple of examples of some of the things that people have built before turning this over to Jeff again.

So here is a very simple script. It's called Amazon light-- simple search. What we did here was, we provided the capability for this one developer to put this little search box over here on his site. And if I do a search for dogs--

[LAUGHTER]

--sorry. So as you'll notice over here on the right-- and this is kind of cool. Over here on the right, when you type in dogs, you will be able to interact with us in a very quick manner. Send a request over to us via XML over HTTP, or via SOAP-- it all depends on the back end system.

That request is then parsed by our back end services. It retrieves information from our back end systems, our databases and our search engine. And then sends back an XML file to this particular website's script. They then use that information and embed it into that existing site.

The very cool thing down here, is it says quick start. And this is where people will be able to configure their website offering. Down here, by answering a couple of these search questions or these questions right here. And then they'll be able to copy the code and just embedded into their site and it auto-magically-- it now allows them to--

[LAUGHTER]

--do search on their own site without having to actually build the whole search engine and the whole technology. You click on that link up here for these top search results and you're brought over to an Amazon site or you're brought over to a Amazon light site that is powered by that third party. Let me go ahead and click on one of the links.

So this will bring you over to Amazon. Now the reason-- or why in the world would anyone want to embed this on their website? Well, we've tied the web services offering to our Associates Program, which would allow the developer or the website owner to earn a percentage for every item sold on their site. So every time they're bringing a customer over to Amazon, they actually are given the opportunity to earn a referral fee. Anywhere between 5% and 15% of the actual price of the item. So there's a built in model-- a built in reason why people would want to use this.

I know I only have a little bit of time so I'm going to go really fast. So there's also a very interesting implementation called Book Search or Book Watch-- as soon as it opens up. What this site will do, when it actually changes, is it's going to make a call to Google's web services offering and Amazon's web services offering, as well as this other web service offering called Book Watch Plus.

And it'll go-- or On Focus, sorry-- and it'll go through and find various topics all on the internet via various peoples web logs. Then it will call Google and then determine what the subjects are for these people or this particular topic. And then it'll call Amazon and find products related to those books that people are talking about in their weblogs.

So what you have here, is an opportunity to see the item that people are talking about in their weblogs. To see the news items that are all over the internet concerning that particular item. And then to see Amazon's similarity engine being used to provide products that people are also buying on our web site.

So this is an amalgamation-- or a combination-- of three different web services displaying a page so that a customer would have a different user interface that would allow them to find out more information or to have a different customer experience. A better customer experience for the things that they're interested in. This is the "surprise us" type of thing. This is what we were very interested in seeing-- people coming up with new ways of consuming our web services offerings to do things that we had not even thought of ourselves.

Then the last one is my favorite. And as Jeff said before-- well actually, this isn't the last, I'm going to do one more. But as Jeff said before, we did something very interesting called an I-mode launch in Japan, which was for our device access group. That's also a group that I manage.

And here in the US, we found a company who knows what's playing on any radio station all across the US. Now they have a web service but they don't have products and they don't have a business model for selling those products. So what did they do?

They actually came up with another build-a-link or build-a-script type of interface where individuals are able to enter a city, a frequency, choose a design-- this is actually what would appear on any website-- enter in an associate ID. That is that ID I was telling you about before that would allow us to track a purchase made and then give credit to that particular associate.

Now the very interesting thing about this is, if I click on-- if I change this to a different station. Can someone give me a Boston station just off the top of their head? Any chance-- any Boston station.

AUDIENCE: MBR.

FREDERICK: MBR, that's a tough one.

AUDIENCE: 92.5.

[INTERPOSING VOICES]

FREDERICK: What was it?

AUDIENCE: 92.5.

FREDERICK: 92.5, thanks. I needed a frequency. MBR was 88.5-- 88.1, yeah that's right. It's been a while. This one's taking a little bit of time. Wouldn't you know, when you doing a demo. But one of the things that I was trying to say here, is that this particular implementation is very unique. This is, yet again, another business model that we had not thought of. This one provider, this yes.net-- whenever it does come up here--

BEZOS: This slow web service is not our web service.

FREDERICK: Exactly.

[LAUGHTER]

BEZOS: We've got some standards.

[LAUGHTER]

FREDERICK: Right. This particular web service offering-- and I don't know why it's not coming up right now-- they came up with a model where they give away their script for free. And one out of five links on the page will be-- they will replace that person's associate ID with their own associate ID. That way they're able to earn that revenue.

BEZOS: So they get 20% of the--

FREDERICK: Exactly.

BEZOS: --revenue share.

FREDERICK: And it's shared all across the internet, where people who have web sites are able to embed this onto their web-- onto their own web sites and not even have to worry about paying for it because it's already built into the system. Sorry here.

AUDIENCE: [INAUDIBLE]

FREDERICK: Oh, it was? Okay. This is their little detail page where you're able to see songs with clips, people who listen to this song also listen to these songs.

[LAUGHTER]

Isn't that cool?

[LAUGHTER]

I really like that one. And then here are a few-- here are some links over to our web site. So if you want to see the last few songs that were played, you can see the song name and then buy that album from amazon.com.

Very last thing, I promise it will be the last. Is a whole store-- a whole store different from Amazon-- built using our web services offering. By a 21-year-old German student who just wanted to play with it and see what he can build. He was able to build this simple camera shop in a matter of-- I believe three hours. That's what he said. And he did something over here where you're able-- great.

[LAUGHTER]

Well anyway, what I was going to show was-- are you sure that I have a connection here? But what he has at this point, is the ability to compare products. This is what this line over here is. And if you would press this Go button and then the next page would allow you to see a comparison across all the different products with a check mark. I just wanted to show that. Hopefully we have we still have a connection.

Let me bring it back over to Amazon. So one of the really cool things about doing web services for amazon.com-- this is looking better-- is that we have a competition going on right now for developers, students, people all across the country to be involved in. Where we're giving away $5,000 to the best storefront builder developed by a developer, or student, or whatever the case may be.

I have some more information about that. So if anyone afterwards would be interested in learning more about the competition, about Amazon Web Services, about device access or just anything at all, just come and see me. Thank you.

BEZOS: Thank you, Robert, very nice.

[APPLAUSE]

FREDERICK: There is one more thing that I wanted to do. And I wanted to recognize some of the people who we have given offers to. We are still hiring, of course. One of the-- let me just-- I don't know if everyone on the list is here today but these are some of your classmates who have job opportunities with us and hopefully they'll join. Alan McConnell, is he here by any chance? Alex Denue, Joe Hastings, Andrew Sutherland, Jonathan Brunsman, and Nicholas Hansons. So I don't know if you know these people, but we're very excited about them having the opportunity to come join our company.

BEZOS: Thank you.

[APPLAUSE]

BEZOS: If you win the web developers contest then you can have a Segway.

[LAUGHTER]

FREDERICK: Is it $5,000?

BEZOS: It's $5,000. Perfect. All right I'm just going to show you a few things. First I'll show you-- I've been talking a lot about personalization. Some of the most impressive things that our architects and engineers have done actually doesn't demo well because a lot of it has to do with scalability and what happens if a particular service goes down. You know, how does it fail gracefully and some of these things are very complicated to do, but not that interesting to demo.

But if I come in-- let me look at my recommendations for a second. And I'll tell you a story the very first time-- now it wasn't the first time-- but one of the first times that I demoed recommendations live-- with a live web site-- I was speaking to a group of Wall Street analysts. My number one recommendation-- number one personal recommendation was a DVD called Slave Girls Beyond Infinity.

[LAUGHTER]

Which was-- this is much less embarrassing and you never know what you're going to get. I never know exactly what I'm going to get because it is live and it does change from minute to minute but I'm glad to see that today I've gotten Apollo.

[LAUGHTER]

And then I can see, of course, why was I recommended this? Because I bought Cosmos and Angle of Attack and another book called Apollo. So Slave Girls Beyond Infinity-- I'll just show you-- was a little embarrassing.

[LAUGHTER]

What happened to that laser pointer thing-- oh here it is. Do not look directly into me-- okay. Here it is, Slave Girls From Beyond Infinity. Let's go over here. This is pretty interesting. So you can buy this DVD with Assault of the Killer Bimbos.

[LAUGHTER]

Buy together, it's only $19.96. All of that product clustering is done automatically, of course. Come down and you'll notice that customers who bought Slave Girls--

[LAUGHTER]

--also bought Sorority Babes and The Slime Ball Bowl-O-Rama and my personal favorite, Cannibal Women in the Avocado Jungle of Death.

[LAUGHTER]

And there's this interesting thing here. We even know when things are part of a series. And also they bought DVDs from the Barbarian Queen series. So we can click through and look at the Barbarian Queen and there's a customer review here. It says, "For Lana Clarkson fans only. This is Barbarian Queen 2. This is not even Barbarian Queen. It's an incredibly bad film whose only redeeming feature is Lana Clarkson in and out of some skimpy outfits. She's completely gorgeous but even she can't save a disaster of a film." Then here comes the great recommendation, "buy Death Stalker instead."

[LAUGHTER]

AUDIENCE: Everything you had in the previous page, is that based on your inventory? Like what's next to each other in your warehouse?

BEZOS: No.

[LAUGHTER]

In fact-- were you trying to be funny?

[LAUGHTER]

No, I didn't think so. No, it's not it's very interesting. We use a technique called Random Stow. So all of the items in the warehouse are stowed randomly. There is no rhyme or reason.

AUDIENCE: Hashed.

BEZOS: As you-- yeah, they're hashed. So as you walk through, they all have locations. They all have BINs so you know where everything is. But only the computer knows where everything is. As you walk through, it'll be Slave Girls From Beyond Infinity, right next to Dr. Seuss. They'll be side by side.

[LAUGHTER]

And so it does make-- walking through, it is like an incredibly eclectic experience, when you go walk through the fulfillment center. But these similarities are incredibly hard to compute well.

And there's a lot of stuff-- you have to look at how much information theory there is in each-- I mean how much information there is in each purchase. Because you don't want, you know-- if you do this simple, straightforward thing then every book just recommends, oh and customers who bought this also bought Harry Potter. And that's not very interesting. So you have to take that into account. You have to do a lot of things.

And we have to compute millions and millions of these and we have to do it very rapidly. And we like to do it on small machines so we figure out how to do that in only two gigabytes of RAM. So there's a lot of-- there's a lot of cool stuff along that.

There was a great similarity, I remember one time, I was looking at this book and it was a book on zen. And I looked down at the list of similarities-- customers who bought this book on zen also bought-- and there were five or six similarities as they usually are. And five of the six were other books about zen and then the sixth one was How to Have a Clutter Free Desk. And that's the kind of connection that only a computer can make.

I mean no human editor is going to say-- they would always pick six other books on zen. But you know, people who like zen also buy books on how to have a clutter free desk and that comes out. Let's see. I'll show you one of my favorite features. I can show it just by any product. I've bought so many things on the site. But I'm going to show you the OXO Salad Spinner in our kitchen store. Here it is. The OXO Good Grips Salad Spinner. Now, this is interesting for a couple of reasons.

One is that the customer reviews on this product read like a sexual experience. I mean--

[LAUGHTER]

--people love the OXO. People who don't eat salad love the OXO Salad Spinner. That's how potent of a product this is. And we do a couple of things here that are interesting.

One is, you can see here it says, these friends have purchased these items. These are friends of mine. And so as long as we're here, and as long as I've already identified my friends, we might as well-- whenever I stumble across something I get to see which of my friends have bought this thing. By the way, doing that in real time isn't that easy. Remember this is a big decentralized system with thousands of web servers and you don't want to go back.

What happens is, we use a pub/sub technology. And when you first come to our website, the instant you're recognized, we publish a message to all of the different services that subscribe. The ones that subscribe to that message recognize it and they start immediately-- the second you hit our gateway-- they start filling a high speed cache of everything that might be interesting. That you might want to see on this visit to the store. So that's one of the ways that we're able to do millions and millions of visits. Provide this level of personalization and customization and have it not be really, really slow.

The other thing that was very personalized on this page. And this is-- I told you earlier that we very carefully test all of our features. When we do this testing, we have a very simple objective function. It's a little more complicated than this but basically our objective function is, we prefer the thing-- if we're testing feature A against feature B, we like the one that generates more sales. We're pretty simple minded in that regard.

But this, what I'm about to show you, is one of the few features we've ever left on the web site even though with statistical significance, it slightly reduces sales. And it's called the Instant Order Update. So you can see up there, at the very top it says, "Jeff Bezos, you purchased this item on May 16th, 2000." So more than two years ago, I bought the OXO Salad Spinner. Now with salad spinners it doesn't happen very much. But with music CDs and a whole bunch of other products, people forget they've bought them. And they buy them a second time.

[LAUGHING]

So they're like, I don't remember-- they buy that Madonna CD and then they're like-- they get it home or maybe they never notice it. But they buy it accidentally a second time. And all of us do unless you're like incredibly organized or something, it's hard to avoid this. So you know, we're busy. People buy books a second time and so on so on.

So anyway, we thought about this and we thought, well what we can do-- unlike any other store anywhere in the world-- is every time you come to a page, if you've already bought it we'll just tell you. Then you won't buy it a second time. Even though this slightly reduces sales, we get so much positive feedback from this. We get so many email messages to customer service where people said, I was just about to buy that Enya CD a second time--

[LAUGHTER]

--and I didn't. Thank you. It's like they're thanking us, because they didn't buy something.

AUDIENCE: [INAUDIBLE]

What's that?

AUDIENCE: Do you ever return items?

BEZOS: Well you know, the amazing thing is most people don't return things very much. You know what they will do, because it's a bit of a hassle to return something, if they got a second copy of that Enya CD, typically they would give it to a friend or something. They don't really want it but it's not-- for an expensive item, yes people would return it. But people also tend to keep track of their-- they're not going to, oh I already bought a digital camera.

[LAUGHTER]

Oh silly me. That's right I have a Segway!

[LAUGHTER]

Shoot!

[LAUGHTER]

But for a $12 CD, people don't put the same amount of attention on it. So anyway, I'll just show you. This is just amazing to me. Every time I look at this, there are new reviews. So this one was only written on November 9th. And it's from this guy named Warren Holzum, "this is one heck of a spinner. I've darn near gotten up to centrifuge speeds with just a few easy pumps--

[LAUGHTER]

--great for drying off all kinds of fruits and vegetables." I mean just goes on and on. Literally, there are-- well let's see, it should say down here. I don't know how many customers reviews there are. There are hundreds. Well here it only says 74. I don't know why there are only 74-- I thought there were more.

But anyway, there's 74 customer reviews here. Look this one is labeled "Sand," from Georgina. "This is a very good spinner, at a good price. Most of the previous reviews are dead on. Except they neglect to mention that this spinner does a great job of removing sand from red lettuce and other exotic greens. Including dandelion, radish, not to mention spinach. I bought it for that reason alone."

[LAUGHTER]

People are just fanatical salad spinners. Look at this, it's all five star reviews. Oh here's somebody, "good but questionable durability." There's always one in the crowd, here.

[LAUGHTER]

We can sort these reviews. It's not a totally straightforward algorithm, but we can basically-- we try to put the reviews-- if you come back to the main page-- some of the top reviews are from people who've gotten lots of helpful votes. So you can always vote-- was this review helpful to you? Yes or no.

One of the nice things about voting on amazon.com, is we have customer identity because everybody who's using our system has a credit card number and so we can kind of keep track of-- we don't let somebody stack the deck and vote 100 times. I mean I guess it could if they have a 100 different credit card numbers and 100 different accounts but it's not really worth stacking the deck for the salad spinner to get so many credit cards.

AUDIENCE: Unless you make it.

BEZOS: Well I guess that's true, if you make it then why not? If you make it and you're evil.

[LAUGHTER]

We did have-- it was very funny. We do remove them. And I often think we should at least build a graveyard for them somewhere. But we do have people who put up spoof customer reviews and they're hilarious. We've had God review the Bible.

[LAUGHTER]

We had-- I can't remember which Bronte sister it is, the one who really hated Jane Austen. It was either Emily or Charlotte. Does anybody know?

AUDIENCE: Emily.

BEZOS: Is it Emily? So we had Emily-- a few years ago there were like two mini series and a full length motion picture based on different Jane Austen novels. This was just a few years ago. And somebody started posting customer reviews pretending to be Emily Bronte.

And she was just like, I hate Jane Austen. Two mini series in a full length motion picture in one year? They were-- it was quite amusing but we take those down. We also had one guy who said-- his review was incredibly honest, it wasn't a spoof. But it was also kind of like damning with faint praise. His review started out, this is the best book my brother ever wrote.