Jeff Bezos (Amazon.com) - "Earth's Most Customer-Centric Company: Differentiating With Technology”
HOST: So I don't think Jeff Bezos needs much of an introduction. You've perhaps heard of this small company he runs out of some city in the west coast--
--where there's another company. I took a poll of the faculty recently and-- I was at a faculty lunch and we were talking about what people did online. And I sort of ask people questions and eventually I said, well, how many of you have bought something on amazon.com. And it was actually 100%, I think, raised their hand.
BEZOS: Yeah, baby!
HOST: I guarantee it was not one of those.
Any rate, Jeff is a graduate of Princeton. And I'm pleased to say, as he was telling me upstairs, he graduated from Princeton when it was still an EECS-- an electrical engineering and computer science department. Before the department bifurcated. And so he does have an electrical engineering and computer science degree. And I think-- he tells me he's glad he has one of those. And as he should be.
After leaving Princeton, he went to Bankers Trust. He worked there in the development of computer systems. So presumably if you need help with code, he can help you with some of your programming and stuff of that nature.
After that he went to D.E. Shaw. Well known for their high-tech approach to finance. And in fact, one of two financial services companies-- three, I guess-- that belongs to our industrial program in the department. And then he left D.E. Shaw to start amazon.com. And I'm sure he can tell you more about that than I can. Jeff.
BEZOS: Thank you.
Thank you, you guys it's great to be here. So I'll start out telling you a little bit about the founding of amazon.com and then I'm going to do my favorite thing which is to give a live website demo.
This is the house where Amazon was founded. On the left-hand side here is this little enclosed garage. I was living in New York City, working for D.E. Shaw and Co and I came across the fact that web usage was growing at 2,300% a year. This was one of those weird things where you could measure the rate of growth without actually knowing the baseline usage.
And people were doing that because they could put sniffers at various points-- at various nodes-- and they could sort of see how fast the traffic levels were growing past those nodes. And they could get a statistical sample of those but they actually had no idea what baseline usage was.
But you could look around the web already, in the spring of 1994, and see that it was big enough that if it was growing at 2,300% a year, pretty soon it was going to be huge. And so I kind of packed up my wife and I and we flew to Fort Worth, Texas where my dad gave us a car-- a 1988 Chevy Blazer. Which by the way, Consumer Reports recommends not to buy used under any circumstances for any price. And we drove that car from Texas to Seattle.
Seattle was chosen as the best place to start amazon.com because it was a large pool of technical talent and also nearby at the time, was the largest book warehouse in the world. In a town called Roseburg, Oregon.
So we got there and the first order of business was to recruit a V.P. of Engineering. I talked to several people and I found the guy I really liked, his name was Shell. He was with the company for six years and was a huge instrumental force in the company and in the early days-- not single-handedly of course, but in large part-- led all of the software engineering that was amazon.com.
He had nothing to do with the look of this website, though. This is what our website looked like when we launched in 1995. I wrote all of that HTML myself--
--please hold your applause.
But Shell did the hard part, he wrote all of the code that actually made it work. Before hiring Shell, I was making our desks out of doors and four by fours.
It took me three months to convince Shell to come join this company because it was really just a one person company. It was me and it was a piece of paper and a name. And the name wasn't even Amazon that time, it was Cadabra. Cadabra got changed because whenever I would say it to anybody over the phone, they would think I said cadaver--
--and I realized that wasn't going to work very well. Amazon was incorporated, by the way, by I-- I only knew two people in Seattle so we're driving out there, I wanted to be moving fast. I wanted the company be incorporated by the time we arrived.
So I called one of two friends I had in Seattle and said, could you recommend a lawyer who could help me get a company Incorporated and open bank accounts for the company. And do some stuff so I can hit the ground running. My friend said, sure. So amazon.com was incorporated by my friend's divorce lawyer--
--not like some big high-powered attorney.
But anyway, I finally talked Shell into coming and joining this intrepid venture. But before he had moved from the Bay Area up to Seattle-- he was getting ready to do that move. I called him and I said, Shell, how tall do you want your desk to be? Because I'm making them out of doors I bought at Home Depot and four by fours. There was like 10 seconds of silence on the line and I was thinking to myself, oh great, now he realizes what a two-bit operation this really is and he's going to change his mind. He eventually answered the question.
About a year later I said, Shell, do you remember when I asked you how tall you wanted your desk to be a year ago? You were quiet for like 10 seconds, were you like reconsidering your decision? And he said, no I was just considering how tall I wanted my desk to be.
So we got the software put together-- took about a year to get the software put together and to write this astonishingly beautiful HTML. It's really ugly, isn't it?
It is. And if you'll see-- I think the next slide coming up-- no, well this is our next real office. This is what-- I don't have what our website looks like today, but you guys know what our website looks like today.
You will note, that it has some minor improvements like a search box on the main page. Here you had to actually click through to search. You had to click on that link that says, 1 million titles. So things have gotten better.
By the way, since people don't know this, our number one-- the number one thing we've invested in at amazon.com has been technology. And in fact, since 1997, we have-- in the last five years-- we've spent $800 million on technology. So when you look at what it is today and look back at this-- that's what $800 million will buy.
We've only spent $300 million on five million square feet of fulfillment centers, for example. And only $600 million on marketing. So technology has been our single biggest investment and the company basically runs on computer science. It's either that or it's just my bias since that's my background. But at any rate, that's what we do. We really focus on trying to differentiate our experience with the engineering and the algorithms that we put into it.
So anyway, we got this company just about ready to launch and we looked at it and looking at this little tiny fulfillment center-- we were here by then, in this building. We had a basement fulfillment center. To call it a fulfillment center was very grand. There's a lot of puffery. It was 400 square feet, which is about the size of a one car garage.
We launched this business and we're looking at it and we didn't know if anybody would order from us. We really didn't. And in fact, one of the software engineers looked at this little space and he said, I can't figure out whether this is incredibly optimistic or hopelessly pathetic.
Indeed, we didn't know. We had no idea whether anybody would want to buy things in this way. The business plan called for generating sales very slowly as customers change their attitudes. The original business plan-- which I thought was very optimistic at the time-- called for amazon.com to generate $70 million in sales in the year 2001. We actually generated in excess of $3 billion in sales in the year 2001.
We knew, by the way, that we were really on to something in those first 30 days. In the first 30 days we got orders from all 50 states and 45 different countries. With not a dollar of advertising, just all word of mouth. In the first year we didn't spend anything on advertising. All of it was word of mouth. And that really forged the company. We were going to focus on customer experience. Because we saw the power of word of mouth so very, very clearly in those early days.
Something great about word of mouth online, by the way-- and feedback from customers-- which is that email turns off the politeness gene in the human being. It's wonderful. So people actually tell you what they really think. Usually in all caps.
So we get lots of info and feedback from our customers and we try to use it well.
Well, in that first few months, we got one order-- it was literally in the first six months, I remember exactly when it was-- an order from Bulgaria. I didn't even know they had internet access in 1995 in Bulgaria, but they did. This person did not pay with a credit card. They paid with cash. Which is a method of payment we discourage.
They had taken two crisp $100 bills and folded them up in a tiny little package. And they took those $100 bills and put them inside a floppy disk. They opened the little metal door and slipped them in the floppy disk. And then they mailed us the floppy disk.
And on the note-- on the floppy disk was a little note and it said, the money is inside the floppy disk.
And then it went on to say, the customs inspectors steal the money, but they don't read English. So we opened up the little door sure enough, there was $200 in there. They had written their order number on it so we could associate it with-- and we shipped them their books and all was well.
Our vision has changed a little bit but it hasn't changed in about four years. About four years ago, we came up with the notion that we wanted to be a place-- build a place where people can come to find and discover anything that they might want to buy online. Literally universal selection. Anything with a capital "A."
We had already been on the path of trying to be what we call, Earth's most customer centric company. And we mean three things. We have a very precise definition for what customer centricity is at amazon.com. It means listen, invent, and personalize.
The basic idea is you can't really run a business of any kind, I think in most people's opinion, if you're not going to listen to your customers. That sort of fundamental. But invention is equally important. And invention, it goes far beyond listening to customers. Because oftentimes, customers don't really know what it is that they want. It's your job to invent on their behalf and we've done a tremendous amount of invention and innovation of different kinds in the company. I'll tell you a little bit more about that when I give you the live demo.
And then finally personalization. The notion of customer centricity is to build a place where each individual customer has his or her own website. And to do that in large part, not by explicit information that the customer provides, but through implicit observation of their behavior on the website. So that we personalize that experience based on the actions that they take.
And we put a lot of effort into that. We've been working on that for-- well for all but for the first year of the company's operations. So for, I guess about six years now. A lot of-- some of our smartest people have worked on that. There are bunch of areas where good algorithms, good heuristics, and so on, turn out to be very important for driving our customer experience.
And that kind of discovery-- accelerating discovery for customers, which is what personalization is largely all about, is critical when you use this. When you decide to be selection intensive. So we have over 28 million, or something like that, items in our catalog. We have-- across all categories.
When you have so many items, you have to work hard to build tools that help customers find products. But you also have to work hard at something which is a little less intuitive and actually, technically, even more challenging. Which is to help products find customers. So we reverse the sense of that. We've put a lot of energy into helping products find customers and it's paid off.
This is the basic thing that we work on every day. So what we want to do is have the world's best customer experience. And there are a few things that feed into that. Massive selection-- which requires great discovery tools-- and low prices. Those are the two biggest things.
Low prices-- and both of those things-- the heart of both things is technology. It's very interesting because I often get asked-- and again maybe it's just my bias because I'm an EECS geek-- but how our business differs from traditional retail.
The answer to that question-- I happen to know the guy who does real estate for Starbucks. And believe me, this is like the guy who has the metaphorical corner office. I mean, that business is all about real estate. This is why the old retail saw-- the three most important things in retail are location, location, location. And in our business it's technology, technology, technology.
And one of the great things about technology as opposed to real estate, is that real estate on average-- economists will tell you-- will get more expensive at the rate of inflation. So real estate gets more expensive every year. And that's the core ingredient of physical retailing.
Whereas our core ingredients get to follow Moore's law. So disk space getting twice as cheap every 12 months. Bandwidth getting twice as cheap about at that rate. CPU every 18 months. This is a tremendous advantage.
As those things get cheaper and cheaper and cheaper-- in and of themselves, they don't offer any real benefit to customers. But what we get to do is take all that compute power and layer innovation on top of it to really change the customer experience game. And that's a lot of fun.
We have invested enough in our software that we've become the more-- a big part of what we've done is become more classic e-commerce company. A more classic technology company, too. Where we actually now service-- make available as an application service provider-- is an ASP model, our e-commerce technology-- to others. We also have done-- and I think Rob is going to give you guys a quick demo of our web services stuff in a minute, too. Are we still doing that competition?
BEZOS: Oh, good. This is what-- instead of our 400 square foot fulfillment center, we now have five million square feet of full of fulfillment space. It turns out that operations research and all the logistic stuff we do here is astonishingly similar. When I started learning about these techniques I was like, wait a second this is computer science, they just have different words for everything.
One of the things that's very interesting is, in some of these fulfillment centers, we have two million different items in the fulfillment center. And you can order any two of those and somehow, we have to marry those two items together and get them into a single box and ship them to you. That sortation is very challenging and there are many pieces of it that are challenging-- to do in an efficient way.
All of the most obvious things you would try first turn out-- they work but they don't work efficiently. The cost would be too high. And what you end up doing is, using very sophisticated algorithms to calculate an optimal pick path through the fulfillment center.
you pick a bunch of customer orders at once and a decentralized swarm of people actually pick all these things. And then they have to get married and it's sort of a two stage sort later. But the picking algorithms are like a dynamic traveling salesman problem where the cities occasionally move and disappear. It's like unbelievably--
--it's actually very fun. And I didn't know-- when I started the business, I thought the front end of the business would be very fun and intellectually stimulating and had envisioned a lot of the kind of personalization features that we could build online. But one of the things that I had not known and not envisioned, is how intellectually stimulating inventory optimization and pick path optimization and these kinds of things would be, too. This is kind of like airline routing and so on. It's a very interesting problem.
AUDIENCE: You wrote your own software for that?
BEZOS: We wrote our own software for that. So it's a great question. In fact, we bought off-the-shelf inventory optimization software. There is no good picking software for the two million items so that we built from the very beginning. We did buy inventory optimization software and it didn't work very well for such a large SKU base.
One of the things that we found, is that our operations are significantly different from most companies for a variety of reasons. Your typical mail order company-- most of these things are made for like warehouse management system and made for mail order companies. And you can buy the things off the shelf but they typically have a different number of SKUs measured in the tens of thousands. And a lot of things change when you scale that up to millions of SKUs instead of tens of thousands of SKUs. And so we had to do a lot of it ourselves.
We bought-- early on we bought some personalization software that also didn't work. And one of the things that's interesting about personalization-- and that's where we put a huge amount of energy and attention-- is that first of all, it has to work at scale with millions of customers. And on peak days we will take orders for and ship 2 million items. And so it's very, very sensitive to doing that-- and we do for unrecognized customers, we do real-time personalization based on their click stream. So there really is nobody doing that.
The other thing is, that we can do active experiments. This is like a really big point that most people miss. But it's one thing to take a huge data set and sort of do backward looking data mining on it. And you can actually find some interesting things that way. But what you can't get from that, is anything that has a feedback loop. Where if you change something, changing that thing is actually going to change the customer behavior.
So one of the most fascinating tools we have at our disposal, is the ability to do active experiments. It's kind of this huge laboratory with literally, 30 million visitors a month coming through. And every time we make a little change on the website, we roll that change out to a randomly selected group of people and hold constant the rest.
So we call it A/B testing. We may do-- half the people see version A and half see version B. They're seeing it simultaneously, so we're holding all the factors constant. And we can actually tell. So if we develop a new personalization algorithm that some smart person inside the company comes up with, and they're really jazzed and they think it's better, we never have to argue about that. We say, well let's try it. And a few hours later, we will know whether it's better. And that is huge because those kinds of arguments are so useless and energy draining.
Come on up. I'm going to introduce-- this is Robert. You're an alum.
FREDERICK: Yes I am.
BEZOS: And Robert has been with us for quite a while. How long have you been?
FREDERICK: About 3 and 1/2 years.
BEZOS: 3 and 1/2 years. We've been to Japan together. We've been through several trials together. All good things. You've worked on our wireless business and now you're doing a lot with web services. Let me turn it over to you.
FREDERICK: So again, my name is Robert Frederick. I manage amazon.com's web services program. How many of you have heard of web services? Or are doing things with web services? Okay, great. Amazon Web Services is something that we launched this year in July. We found that by providing an API-- a solid SDK or an API-- to our broad website developers as well as our associates, that we would be able to get more and more users to interact with our features all over the internet rather than just Amazon's website.
So what we were thinking of doing is, we had three goals. We wanted to define a product. And that product is the ability to create your own store on your own website. To expose Amazon services and features on, I believe it at this point in time, it's 900,000 websites that interact with this on a day by day basis.
So what we were doing is, we were defining a capability to interact. To get search results. To interact with our similarities engine. To find products and to build those storefronts on external websites all over the internet in various countries. All through exposing a solid API and SDK.
So again, we had a market of 900,000 websites. We have developers who wanted to build SOAP packages, Scripts, Perl, PHP, you name it-- .net applications. And we wanted to provide those applications to each one of these web site owners so that they can use our tools and interact with us via soap or via XML over HTTP, in a very easy and viable solution.
So the economic model for this as well-- or why are we doing this? Why would Amazon ever want to expose the services that we have on our site to all of these other websites on the internet? And the reason is very simple. The more people that are interacting with our products, the more items that will potentially sell.
And then we also have the capability to provide these products on their website in a way that doesn't have to make them recreate the wheel. It's easy to implement. Literally, anywhere between five minutes to a couple of hours, and people are able to have a very robust storefront. A very robust way of interacting with our products and our selection.
What we ended up doing was, we came up with a SOAP interface and a XML over HTTP interface. And I just wanted to go through a couple of examples of some of the things that people have built before turning this over to Jeff again.
So here is a very simple script. It's called Amazon light-- simple search. What we did here was, we provided the capability for this one developer to put this little search box over here on his site. And if I do a search for dogs--
--sorry. So as you'll notice over here on the right-- and this is kind of cool. Over here on the right, when you type in dogs, you will be able to interact with us in a very quick manner. Send a request over to us via XML over HTTP, or via SOAP-- it all depends on the back end system.
That request is then parsed by our back end services. It retrieves information from our back end systems, our databases and our search engine. And then sends back an XML file to this particular website's script. They then use that information and embed it into that existing site.
The very cool thing down here, is it says quick start. And this is where people will be able to configure their website offering. Down here, by answering a couple of these search questions or these questions right here. And then they'll be able to copy the code and just embedded into their site and it auto-magically-- it now allows them to--
--do search on their own site without having to actually build the whole search engine and the whole technology. You click on that link up here for these top search results and you're brought over to an Amazon site or you're brought over to a Amazon light site that is powered by that third party. Let me go ahead and click on one of the links.
So this will bring you over to Amazon. Now the reason-- or why in the world would anyone want to embed this on their website? Well, we've tied the web services offering to our Associates Program, which would allow the developer or the website owner to earn a percentage for every item sold on their site. So every time they're bringing a customer over to Amazon, they actually are given the opportunity to earn a referral fee. Anywhere between 5% and 15% of the actual price of the item. So there's a built in model-- a built in reason why people would want to use this.
I know I only have a little bit of time so I'm going to go really fast. So there's also a very interesting implementation called Book Search or Book Watch-- as soon as it opens up. What this site will do, when it actually changes, is it's going to make a call to Google's web services offering and Amazon's web services offering, as well as this other web service offering called Book Watch Plus.
And it'll go-- or On Focus, sorry-- and it'll go through and find various topics all on the internet via various peoples web logs. Then it will call Google and then determine what the subjects are for these people or this particular topic. And then it'll call Amazon and find products related to those books that people are talking about in their weblogs.
So what you have here, is an opportunity to see the item that people are talking about in their weblogs. To see the news items that are all over the internet concerning that particular item. And then to see Amazon's similarity engine being used to provide products that people are also buying on our web site.
So this is an amalgamation-- or a combination-- of three different web services displaying a page so that a customer would have a different user interface that would allow them to find out more information or to have a different customer experience. A better customer experience for the things that they're interested in. This is the "surprise us" type of thing. This is what we were very interested in seeing-- people coming up with new ways of consuming our web services offerings to do things that we had not even thought of ourselves.
Then the last one is my favorite. And as Jeff said before-- well actually, this isn't the last, I'm going to do one more. But as Jeff said before, we did something very interesting called an I-mode launch in Japan, which was for our device access group. That's also a group that I manage.
And here in the US, we found a company who knows what's playing on any radio station all across the US. Now they have a web service but they don't have products and they don't have a business model for selling those products. So what did they do?
They actually came up with another build-a-link or build-a-script type of interface where individuals are able to enter a city, a frequency, choose a design-- this is actually what would appear on any website-- enter in an associate ID. That is that ID I was telling you about before that would allow us to track a purchase made and then give credit to that particular associate.
Now the very interesting thing about this is, if I click on-- if I change this to a different station. Can someone give me a Boston station just off the top of their head? Any chance-- any Boston station.
FREDERICK: MBR, that's a tough one.
FREDERICK: What was it?
FREDERICK: 92.5, thanks. I needed a frequency. MBR was 88.5-- 88.1, yeah that's right. It's been a while. This one's taking a little bit of time. Wouldn't you know, when you doing a demo. But one of the things that I was trying to say here, is that this particular implementation is very unique. This is, yet again, another business model that we had not thought of. This one provider, this yes.net-- whenever it does come up here--
BEZOS: This slow web service is not our web service.
BEZOS: We've got some standards.
FREDERICK: Right. This particular web service offering-- and I don't know why it's not coming up right now-- they came up with a model where they give away their script for free. And one out of five links on the page will be-- they will replace that person's associate ID with their own associate ID. That way they're able to earn that revenue.
BEZOS: So they get 20% of the--
BEZOS: --revenue share.
FREDERICK: And it's shared all across the internet, where people who have web sites are able to embed this onto their web-- onto their own web sites and not even have to worry about paying for it because it's already built into the system. Sorry here.
FREDERICK: Oh, it was? Okay. This is their little detail page where you're able to see songs with clips, people who listen to this song also listen to these songs.
Isn't that cool?
I really like that one. And then here are a few-- here are some links over to our web site. So if you want to see the last few songs that were played, you can see the song name and then buy that album from amazon.com.
Very last thing, I promise it will be the last. Is a whole store-- a whole store different from Amazon-- built using our web services offering. By a 21-year-old German student who just wanted to play with it and see what he can build. He was able to build this simple camera shop in a matter of-- I believe three hours. That's what he said. And he did something over here where you're able-- great.
Well anyway, what I was going to show was-- are you sure that I have a connection here? But what he has at this point, is the ability to compare products. This is what this line over here is. And if you would press this Go button and then the next page would allow you to see a comparison across all the different products with a check mark. I just wanted to show that. Hopefully we have we still have a connection.
Let me bring it back over to Amazon. So one of the really cool things about doing web services for amazon.com-- this is looking better-- is that we have a competition going on right now for developers, students, people all across the country to be involved in. Where we're giving away $5,000 to the best storefront builder developed by a developer, or student, or whatever the case may be.
I have some more information about that. So if anyone afterwards would be interested in learning more about the competition, about Amazon Web Services, about device access or just anything at all, just come and see me. Thank you.
BEZOS: Thank you, Robert, very nice.
FREDERICK: There is one more thing that I wanted to do. And I wanted to recognize some of the people who we have given offers to. We are still hiring, of course. One of the-- let me just-- I don't know if everyone on the list is here today but these are some of your classmates who have job opportunities with us and hopefully they'll join. Alan McConnell, is he here by any chance? Alex Denue, Joe Hastings, Andrew Sutherland, Jonathan Brunsman, and Nicholas Hansons. So I don't know if you know these people, but we're very excited about them having the opportunity to come join our company.
BEZOS: Thank you.
BEZOS: If you win the web developers contest then you can have a Segway.
FREDERICK: Is it $5,000?
BEZOS: It's $5,000. Perfect. All right I'm just going to show you a few things. First I'll show you-- I've been talking a lot about personalization. Some of the most impressive things that our architects and engineers have done actually doesn't demo well because a lot of it has to do with scalability and what happens if a particular service goes down. You know, how does it fail gracefully and some of these things are very complicated to do, but not that interesting to demo.
But if I come in-- let me look at my recommendations for a second. And I'll tell you a story the very first time-- now it wasn't the first time-- but one of the first times that I demoed recommendations live-- with a live web site-- I was speaking to a group of Wall Street analysts. My number one recommendation-- number one personal recommendation was a DVD called Slave Girls Beyond Infinity.
Which was-- this is much less embarrassing and you never know what you're going to get. I never know exactly what I'm going to get because it is live and it does change from minute to minute but I'm glad to see that today I've gotten Apollo.
And then I can see, of course, why was I recommended this? Because I bought Cosmos and Angle of Attack and another book called Apollo. So Slave Girls Beyond Infinity-- I'll just show you-- was a little embarrassing.
What happened to that laser pointer thing-- oh here it is. Do not look directly into me-- okay. Here it is, Slave Girls From Beyond Infinity. Let's go over here. This is pretty interesting. So you can buy this DVD with Assault of the Killer Bimbos.
Buy together, it's only $19.96. All of that product clustering is done automatically, of course. Come down and you'll notice that customers who bought Slave Girls--
--also bought Sorority Babes and The Slime Ball Bowl-O-Rama and my personal favorite, Cannibal Women in the Avocado Jungle of Death.
And there's this interesting thing here. We even know when things are part of a series. And also they bought DVDs from the Barbarian Queen series. So we can click through and look at the Barbarian Queen and there's a customer review here. It says, "For Lana Clarkson fans only. This is Barbarian Queen 2. This is not even Barbarian Queen. It's an incredibly bad film whose only redeeming feature is Lana Clarkson in and out of some skimpy outfits. She's completely gorgeous but even she can't save a disaster of a film." Then here comes the great recommendation, "buy Death Stalker instead."
AUDIENCE: Everything you had in the previous page, is that based on your inventory? Like what's next to each other in your warehouse?
In fact-- were you trying to be funny?
No, I didn't think so. No, it's not it's very interesting. We use a technique called Random Stow. So all of the items in the warehouse are stowed randomly. There is no rhyme or reason.
BEZOS: As you-- yeah, they're hashed. So as you walk through, they all have locations. They all have BINs so you know where everything is. But only the computer knows where everything is. As you walk through, it'll be Slave Girls From Beyond Infinity, right next to Dr. Seuss. They'll be side by side.
And so it does make-- walking through, it is like an incredibly eclectic experience, when you go walk through the fulfillment center. But these similarities are incredibly hard to compute well.
And there's a lot of stuff-- you have to look at how much information theory there is in each-- I mean how much information there is in each purchase. Because you don't want, you know-- if you do this simple, straightforward thing then every book just recommends, oh and customers who bought this also bought Harry Potter. And that's not very interesting. So you have to take that into account. You have to do a lot of things.
And we have to compute millions and millions of these and we have to do it very rapidly. And we like to do it on small machines so we figure out how to do that in only two gigabytes of RAM. So there's a lot of-- there's a lot of cool stuff along that.
There was a great similarity, I remember one time, I was looking at this book and it was a book on zen. And I looked down at the list of similarities-- customers who bought this book on zen also bought-- and there were five or six similarities as they usually are. And five of the six were other books about zen and then the sixth one was How to Have a Clutter Free Desk. And that's the kind of connection that only a computer can make.
I mean no human editor is going to say-- they would always pick six other books on zen. But you know, people who like zen also buy books on how to have a clutter free desk and that comes out. Let's see. I'll show you one of my favorite features. I can show it just by any product. I've bought so many things on the site. But I'm going to show you the OXO Salad Spinner in our kitchen store. Here it is. The OXO Good Grips Salad Spinner. Now, this is interesting for a couple of reasons.
One is that the customer reviews on this product read like a sexual experience. I mean--
--people love the OXO. People who don't eat salad love the OXO Salad Spinner. That's how potent of a product this is. And we do a couple of things here that are interesting.
One is, you can see here it says, these friends have purchased these items. These are friends of mine. And so as long as we're here, and as long as I've already identified my friends, we might as well-- whenever I stumble across something I get to see which of my friends have bought this thing. By the way, doing that in real time isn't that easy. Remember this is a big decentralized system with thousands of web servers and you don't want to go back.
What happens is, we use a pub/sub technology. And when you first come to our website, the instant you're recognized, we publish a message to all of the different services that subscribe. The ones that subscribe to that message recognize it and they start immediately-- the second you hit our gateway-- they start filling a high speed cache of everything that might be interesting. That you might want to see on this visit to the store. So that's one of the ways that we're able to do millions and millions of visits. Provide this level of personalization and customization and have it not be really, really slow.
The other thing that was very personalized on this page. And this is-- I told you earlier that we very carefully test all of our features. When we do this testing, we have a very simple objective function. It's a little more complicated than this but basically our objective function is, we prefer the thing-- if we're testing feature A against feature B, we like the one that generates more sales. We're pretty simple minded in that regard.
But this, what I'm about to show you, is one of the few features we've ever left on the web site even though with statistical significance, it slightly reduces sales. And it's called the Instant Order Update. So you can see up there, at the very top it says, "Jeff Bezos, you purchased this item on May 16th, 2000." So more than two years ago, I bought the OXO Salad Spinner. Now with salad spinners it doesn't happen very much. But with music CDs and a whole bunch of other products, people forget they've bought them. And they buy them a second time.
So they're like, I don't remember-- they buy that Madonna CD and then they're like-- they get it home or maybe they never notice it. But they buy it accidentally a second time. And all of us do unless you're like incredibly organized or something, it's hard to avoid this. So you know, we're busy. People buy books a second time and so on so on.
So anyway, we thought about this and we thought, well what we can do-- unlike any other store anywhere in the world-- is every time you come to a page, if you've already bought it we'll just tell you. Then you won't buy it a second time. Even though this slightly reduces sales, we get so much positive feedback from this. We get so many email messages to customer service where people said, I was just about to buy that Enya CD a second time--
--and I didn't. Thank you. It's like they're thanking us, because they didn't buy something.
AUDIENCE: Do you ever return items?
BEZOS: Well you know, the amazing thing is most people don't return things very much. You know what they will do, because it's a bit of a hassle to return something, if they got a second copy of that Enya CD, typically they would give it to a friend or something. They don't really want it but it's not-- for an expensive item, yes people would return it. But people also tend to keep track of their-- they're not going to, oh I already bought a digital camera.
Oh silly me. That's right I have a Segway!
But for a $12 CD, people don't put the same amount of attention on it. So anyway, I'll just show you. This is just amazing to me. Every time I look at this, there are new reviews. So this one was only written on November 9th. And it's from this guy named Warren Holzum, "this is one heck of a spinner. I've darn near gotten up to centrifuge speeds with just a few easy pumps--
--great for drying off all kinds of fruits and vegetables." I mean just goes on and on. Literally, there are-- well let's see, it should say down here. I don't know how many customers reviews there are. There are hundreds. Well here it only says 74. I don't know why there are only 74-- I thought there were more.
But anyway, there's 74 customer reviews here. Look this one is labeled "Sand," from Georgina. "This is a very good spinner, at a good price. Most of the previous reviews are dead on. Except they neglect to mention that this spinner does a great job of removing sand from red lettuce and other exotic greens. Including dandelion, radish, not to mention spinach. I bought it for that reason alone."
People are just fanatical salad spinners. Look at this, it's all five star reviews. Oh here's somebody, "good but questionable durability." There's always one in the crowd, here.
We can sort these reviews. It's not a totally straightforward algorithm, but we can basically-- we try to put the reviews-- if you come back to the main page-- some of the top reviews are from people who've gotten lots of helpful votes. So you can always vote-- was this review helpful to you? Yes or no.
One of the nice things about voting on amazon.com, is we have customer identity because everybody who's using our system has a credit card number and so we can kind of keep track of-- we don't let somebody stack the deck and vote 100 times. I mean I guess it could if they have a 100 different credit card numbers and 100 different accounts but it's not really worth stacking the deck for the salad spinner to get so many credit cards.
AUDIENCE: Unless you make it.
BEZOS: Well I guess that's true, if you make it then why not? If you make it and you're evil.
We did have-- it was very funny. We do remove them. And I often think we should at least build a graveyard for them somewhere. But we do have people who put up spoof customer reviews and they're hilarious. We've had God review the Bible.
We had-- I can't remember which Bronte sister it is, the one who really hated Jane Austen. It was either Emily or Charlotte. Does anybody know?
BEZOS: Is it Emily? So we had Emily-- a few years ago there were like two mini series and a full length motion picture based on different Jane Austen novels. This was just a few years ago. And somebody started posting customer reviews pretending to be Emily Bronte.
And she was just like, I hate Jane Austen. Two mini series in a full length motion picture in one year? They were-- it was quite amusing but we take those down. We also had one guy who said-- his review was incredibly honest, it wasn't a spoof. But it was also kind of like damning with faint praise. His review started out, this is the best book my brother ever wrote.
I'm like, golly that's a tough family.
Unlike all my other brothers books this one might be worth reading.
So let's see, I don't know I might just switch over to Q&A. You know, once you go in I can show you the page you made. This is the real time click stream stuff. It's more useful for customers who we don't recognize. But it's you know, trying to make sense of these bizarre pages that we've looked at-- The Dog Listener, Slave Girls, and the OXO salad spinner. We're not giving it much to work with here.
But it's recommending the Microplane Greater Zester. Which by the way, I have that product. That's a great product. And it's recommending Assault of the Killer Bimbos, the Dog Whisperer. It's also recommending the mini salad and herb spinner. So if you need a herb spinner in addition to your salad spinner. But I think I will end the demo there. Let me see if I can get back. I don't even really need this.
AUDIENCE: So these recommendations, they're all based completely on the customer experience? There's no variable of how much money you make?
BEZOS: The customer recommendations are based on-- what we're trying to do is get the probability. What we want is to increase the probability that somebody will add an item to their order, even if it's an inexpensive item. So it actually turns out to work in our interest.
So what we're trying to do is, we look at-- for personalization we basically calculate for every customer, a probability that they will-- what's the most likely product that you will buy. And then we try to take out the most obvious sort of low information content products like Harry Potter. Because probably, you've already run across that 100 times in other ways. In the supermarket and everywhere else. So putting that in front of you, it's not like you're going to say, oh Harry Potter 4 is out. You know you'd have to be living under a rock.
But we try to get the things that are a little bit more obscure products and then within that we optimize those algorithms. And we're certainly doing it out of our own self-interest. I mean we're not-- there's no question. We are in this business to make money. But the way that we do it, is by trying to help customers make purchase decisions.
And if you can do that-- and in small ways, as I showed you with the instant order update-- we've been willing to accept some features that decrease sales. But that's because we're long term oriented. We believe that if we give up those small amounts of sales today, the people who are reminded not to buy the Enya CD a second time, will think that we're so great that they'll come back and buy more in the future.
So it's a very simple model, which is if you treat customers right try to do the right thing and actually provide real value, great prices-- we've been very focused on cost structure. Making the prices-- so that we can afford to lower prices.
So we've got free shipping on orders over $25. 30% off books over $15. Great product prices on electronics and DVD players and digital cameras and so on. And we're the only place you can buy the Segway Human Transporter. Now I realize this is a little out of reach of your average undergrad's budget but nevertheless, it's a cool, cool product.
And then I just like to offer up my email address. I won't answer your email if you email me, but I will forward it. I do read my emails and if anybody is interested, we are always looking for the best of the best in terms of engineering talent, computer science talent. So if you have any interest at all know you can email me, and I'll ship it off right away to the right people.
Anyway with that, I will end and open it to questions and however much time we have, I'll leave that up to you. Who's got a question? Yes, please.
AUDIENCE: There's an interesting controversy a year or two ago, something about different customers were seeing different prices on the same product [INAUDIBLE].
AUDIENCE: I wanted to know what going on, on the business side of that with decisions on what to do.
BEZOS: We did an A/B test with best selling DVDs. Where what we did was, we took-- what we wanted to determine was the shape of the price elasticity in best selling DVDs. So what we did was, we randomly took-- we actually subdivided customers into three groups just based on the last digit of their customer ID. So it was completely random. And we showed some people 30% off, some people 35% off, and some people 40% off.
But what happened is, the press got this all confused and they believed that there was some devious algorithm where we were trying to charge people in rich zip codes more. Which we weren't trying to do. We were actually just trying to determine the shape of the elasticity curve. But it was so confusing for people that we decided not to run that kind of experiment. So we still do all the A/B tests but we do them with features. We don't do them with prices. Yes?
AUDIENCE: More about leadership and process question on how you organize Amazon. Obviously, a lot of the things that you know are very simple basics, like treat your customers well. And then you develop all these great innovations. Now I'm wondering, is it all internal and customer based or do you benchmark against traditional companies and how do you keep this spirit of innovation alive?
BEZOS: Well that is a truly great question. First of all, the way to get a lot of innovation in a company, in my opinion is to try and-- is to work very, very hard to reduce the cost of doing experiments. Because the problem is, if experiments are expensive then very few people are going to get to do very few experiments. That's just the way it works.
So if you have a big group of very creative people-- which we do, we have a tremendous number of smart people who love to innovate on behalf of customers and invent new things. You've got to set up this kind of-- like what I was talking about with the A/B testing framework-- something like that. So that if somebody comes up with a new personalization algorithm that they think will outperform the current one, you don't have to move heaven and earth to get it tried.
So it's important to reduce the cost of experimentation and then to have some objective standards about what's better than what. If you can do that, then you can do a lot of innovation. Now some of the things that we experiment with are really hard to do and the cost of experimentation is high.
For example, we don't really-- and with consumer behavior, it is based-- with anything totally new, it is really very difficult-- anywhere between very difficult and impossible-- to guess how consumers are going to behave. How a mass audience is going to do anything in the future is very hard to determine. The easiest way to determine it, is just to do it and see what happens. And I think a lot of companies get that wrong. They put too much energy into like arguing about how consumers are going to behave and by the time they're done arguing, they could've just done it and seen what happened.
So sometimes-- we spent a huge amount of effort building customer self-service tools. So the customers could cancel their own orders. We did some research up front in the kind of classic way to try and determine whether customers would want to do this. We got some encouraging results but focus groups and things like that are fraught with peril.
People don't know, really typically, what they will do. If you ask them what you think they'll do, they'll try to tell you the truth but they don't really know. And so we decided to just take the gamble. There was no way to build this in a simple way and sort of do A/B test that we could think of. So we just went ahead and built some heavy duty infrastructure to allow people to cancel their own orders.
By the way, internally we argued about, should we let people cancel their own orders? You know these are the kinds of things that you get cold feet about. You're like, maybe they will cancel their orders--
--and maybe we'll have no sales. But in fact, of course, if you give people more control-- you know, our basic theory was, if you give people more control over their environment-- if they know they can cancel their own orders then maybe they'll order more. In fact, that seems to have been correct.
That by giving people these self-service tools, we're empowering them and they seemed to like that. But that was an example where we couldn't really do a low cost experiment. You kind of had to build the whole thing in order to test it. Yes?
AUDIENCE: There was some other less than successful experiment some time ago, when companies were able to look to see from their organization what people were buying. And I'm wondering, with all of the personalization issues, do privacy issues like that crop up?
BEZOS: I'm not sure what you're-- I think you're referring to purchase circles. Which actually has been quite successful. So I'm not sure.
AUDIENCE: Before purchase circles. This was a very long time ago. There was a news story I think on Slashdot, but Xerox had looked to see what its employees were buying.
BEZOS: Yeah but that's still up there. You can do that. No, no, no, no, Xerox can't look to see what their employees are buying. And I don't-- and I doubt Slashdot reported that. What you're thinking of-- I know exactly what you're thinking of. Is that we aggregate the data.
So companies can opt out if they want but I think, like, four companies have opted out. You can go online, I could show you. And you can see what are the best selling books from people who-- at MIT.edu. You can go online and see what are the best selling books from amazon.com employees or from Microsoft employees and so on and so on.
But we have rules about aggregation and we did from the very beginning. So unless the group is at least 250 customers, we don't aggregate that. We don't put the data out there. I think it's 250-- it's some large number. So that people can't back into who bought what. And it turns out to be really cool data, to see what these companies are buying. What the best seller list is.
We do it geographically so you can go in and look at and see what's the best selling book in-- what are the best selling books in Massachusetts. And we do not only best selling, but we do a special algorithm we call, uniquely best selling. Just sort of scrape away all the Harry Potters and so on.
So you can see what's uniquely best selling in Massachusetts. What's uniquely best selling in Boston. What's uniquely best selling in Cambridge. What's uniquely best selling at Harvard and what's uniquely best selling at MIT. And you can compare those and it's actually kind of fun.
And then we put the best seller list-- we tie that back to the detail pages. So if you're stumbling around and you find a detail page, you may see something like, this is the number seven best seller at MIT.edu. Anyway, I think that's what you're thinking of. Yeah?
AUDIENCE: Can you speak back to the decision you made maybe three or four years ago to expand beyond sell books, DVDs and CDs to things like--
BEZOS: Electronics, tools and kitchen.
AUDIENCE: --yeah and one of the questions, I think, of the financial analysts at Amazon would indicate that you don't make any money off anything but books.
BEZOS: Well, but at that time we didn't make money off books either.
The question was, why did we expand into things like electronics, tools, and kitchen since those businesses are not, today, profitable? And the answer is that it increases our addressable market size. So we can ultimately build a much larger company by entering into those new product categories.
And second, while it's true that those new categories, which are only three years old-- and by the way, electronics, tools, and kitchen is now a $600 million business at amazon.com. Which makes it one of the largest. On a standalone basis it's probably in the top four or five e-commerce companies anywhere in the world. So that business is growing and growing very rapidly.
One of the big advantages it has, it has a huge audience of early adopters. Which is why companies like Segway choose us as the exclusive launch platform for products like that. Because really, early adopters are the kind of people who buy stuff like this.
But that business, ultimately, will be-- electronics in particular, in my opinion, will be our largest most profitable business one day. When you start a new business, there are certain fixed costs that come along with starting that business. And then the business has to reach enough scale so that the profit from the business-- the variable profit-- can cover those fixed costs. There's no sidestepping that. It's just a simple investment issue.
The company now, in a trailing 12 month basis, generated $120 million in free cash flow after interest expense. And really we're just at break-even. The company is doing very well financially. Yeah, way in the back.
AUDIENCE: A little bit about-- you were talking about investing in technology. How do you deal with competitors, like Barnes and Noble, for example. They've got that new book browsing thing. How you deal with the fact that new technologies are going out there and how you compete against that, despite the fact that you guys already invest a lot internally?
BEZOS: Well really that is the-- I mean the good news, from a technology investment point of view, is that compared to most of our competitors-- well actually, really, I guess compared to all of our competitors-- we're spending so much more. You would probably have to add up the next 10 e-tailers together to get anywhere near our technology spending budget.
So at $200 million a year-- you know, most of our competitors don't have $200 million in sales. So they can't spend-- we have $4 billion in sales. So $200 million a year is just 5% of sales. At BarnesandNoble.com, $200 million would be half of sales.
So that's a significant advantage for us as we continue to build out. We want the gap between our customer experience and those of our competitors to actually widen. With these kinds of personalization features. There's lots of stuff that I can't talk about on the drawing board, that we think is going to make the experience much, much better in the future.
AUDIENCE: $200 million, how much of it's R&D?
BEZOS: Well, we-- the question was, of that $200 million, how much of it is R&D? I never know exactly how to answer that question. I mean--
AUDIENCE: Capital as opposed to disk drives.
BEZOS: Oh, as opposed to disk drives. Almost all of it is-- I mean there's very little. The good news is, we're a total Linux platform. So it's very cheap. It's all commodity hardware. We have a few big database servers. But the stuff has gotten so cheap. I don't know exactly what the fraction is but a lot of-- a big fraction of that expense is people expense. It's, you know, smart people. Yeah?
AUDIENCE: Who are your competitors? In your state, now where Amazon is, right now?
BEZOS: Well there are two groups of competitors. Our big competition is in the physical world. Online is still so small. Retail sales worldwide are a $5 trillion dollar market. Online is less than a percent of that. It's very, very tiny today. So you don't want to take sales away from online competitors just because there isn't that much sales to take away.
So if you look at physical-- all the sales in the physical world. So that's where the dollars are. That's where you need to focus your primary competitive energy. You look at online companies not because you want to get their sales, but because sometimes you might be able to learn from them. I mean they're watching customers too, hopefully. And so you need to see what they're doing, how are they innovating? What are they thinking of? What kind of lessons can we learn from that and incorporate into doing things the way we do things?
Rarely, by the way, have I seen it successful online to do direct copies of things. Usually, you get some inspiration from something but then you have to make it your own and make it be part of doing things the way we do things for our customers.
Once you're in the physical world, the big competitors-- there's a different one for every category. So we would look at Best Buy in electronics as our top competitor. Best Buy in the physical world. Barnes and Noble in books. And you can go right down the line in every category and you can-- it's usually the kind of the selection intensive superstore type competitor defined by category. Yes?
AUDIENCE: I've got a "as far as the business of web services," question. I've got to imagine that there's a fair amount of money to be made from the information that Amazon generates and is able to collect from consumers.
For example, I know many product designers for baby toys go to Amazon, they look up their customer reviews and figure out how people feel about it. I'm wondering, so now if I were to start using the web services from Amazon to try to resell it, repackage it. How would you guys react? Sort of what's that interplay like?
BEZOS: The answer is, right now you have to try and use the web services as part of our Associates Program. So there's a contract you sign when you sign up to get a developer token. There's some rules. And one of the things you cannot do is download the data and repackage it for the sort of purposes that you're talking about.
And you can't cache the data. There are several things that-- because the data is valuable. We don't want you just to write a program that like sucks all the data out and then it's your data. That can't be.
But you do-- but if you're playing by the rules of being an associate, you can invent a lot of cool stuff. You own that stuff. You read the contract, you own your application and the things you build on top of it. We just reserve the right to take the data back. Yeah? We'll do some more down here in front.
AUDIENCE: So looking to the non-desktop world-- what I guess you could define it--
BEZOS: The non which world?
AUDIENCE: The non-desktop.
AUDIENCE: Mobile or wireless. From a technology perspective, what's uniquely interesting about that world, that makes you want to keep pumping dollars-- or any dollars into it. Or, perhaps is it not unique.
AUDIENCE: So the question is, is the mobile wireless world interesting?
AUDIENCE: Aside from the presentation-- combining lots of things into a small screen.
BEZOS: Right. I think the answer is, that that world is not very interesting today, with the exception of Japan. Where, for whatever reason, it's worked a lot better and people are willing to-- the real problem is the devices.
So the devices have very small input device and very small output device. And as we get to see better web pads-- like this is a great device, the BlackBerry. For doing email this is my most valuable-- I'd give up my cell phone before I would give up my BlackBerry.
You will see devices with this sort of form factor, with decent input devices, that have a decent color screens with enough resolution so you can actually look at enough data to actually do something on the web. Right now, if you're using your cell phone to do things on the web, you're like hitting the three key three times to get a particular-- I mean it's just bad. It's a bad customer experience. It's the best you can do perhaps with that physical device. But that stuff will get better. And that's a pretty big deal.
The thing that's a bigger deal in the shorter term, is more people getting instant on at home. It's not even the extra bandwidth that you get at home. It's the fact that you can leave your computer on. I think the thing that will happen next, is that people will have multiple computers in their home.
So I already see that starting to happen. One of our best selling electronics products for years has been wireless routers. A lot of people installing 802.11 networks in their home. And it completely changes your behavior when you get two or three or four computers in your house instead of through one in the office or one in the bedroom or whatever it is.
When you look at your average household, you're going to start to see that the number of computer-- just like with phones, there was a time when people said, why, would anybody ever have two phones in their house, that's sort of ridiculous. Absurd luxury.
And now, of course, people put a phone in every bedroom. They put them in their bathrooms. They put them in their kitchens. They put them everywhere. And the kitchen is actually the key room. So I put a computer in my kitchen and it doubled my amazon.com purchases.
I strongly recommend-- you need an instant on computer in your kitchen. That will be a big deal.
AUDIENCE: Is your behavior, when you're in the kitchen, different enough that amazon.com would take that into consideration in making your recommendations?
BEZOS: By making kitchen recommendations?
I hadn't thought about that. Recommendations by room. Right. We need to be able to impute your room though. Otherwise we'd have to ask people, what room are you in? Or have you like register your computer. This is my kitchen computer.
We'll make the OXO salad spinner recommendation in there. Yes?
AUDIENCE: A while ago there was a dispute over the Amazon policy on patents, on business methods.
AUDIENCE: What's your feeling on that? Because there was some replication and I believe you made a statement that you thought that it was bad news.
BEZOS: What I think is-- my position on this is very clear which is, I think patents in general are really, very good. If you look at the history of the patent office and of innovation in this country, it's hard to unlink the two.
Now one of the things that I think is-- if you look back historically, the rate of change was slower. So if you look at the lifetime of patents, it's actually increased a little bit because of some international treaties and it's basically gone from 17 years to 20 years.
And for some industries, 20 years is very appropriate lifetime. So for example, if you're manufacturing a new pharmaceutical, you probably need-- you may need 20 years. You've got to go through 10 years of FDA testing, and there's all this stuff going on. You may need a long lifetime to recoup your R&D costs. And you need some exclusive period where all that R&D benefits only you.
But when you look at software patents, it seems to me that 20 years may be too long. And so that's the point I made. It's really the same thing. I would say that there is no chance of that changing. And the reason is, when you go look at this-- and I did, I actually went to Congress and I made my case in front of few people, that seemed like too long a period.
And the problem is, this thing is now so well established, not just in the US but by international treaty. Literally, hundreds of nations that have all entered into mutually binding patent treaties and it's very hard to get that whole web of stuff changed. So that's why it's-- people tend to leave it alone. Yeah.
AUDIENCE: So getting stuff to me. It seems like your business will scale, your computers will scale, but one roadblock between me buying stuff on Amazon all the time, is getting that stuff from Washington to my home. I applaud you all for making that $25 free shipping stuff. I think it's amazing, I don't know how you do it. But where do you think that's going to go? How do you think those costs scale?
BEZOS: Well it's a great question. The question basically is, is there some way to dramatically change the delivery part of the business model, so that you are-- and the answer is, at certain scale levels, maybe. So already it's changed a lot because when we first started, we shipped only out of Seattle. Now we have--
AUDIENCE: Fine, but you're not going to have one in every city.
BEZOS: Well, why not?
AUDIENCE: Do you get the cost benefit from that, though?
BEZOS: Well it's all a question of scale. If the question was, you're not going to have one in every city, then you probably won't have one in every city. But you very well might have one-- I mean, you need a lot of scale. At $4 billion in scale, we've got the right number of fulfillment centers right now. But if you take that from $4 billion to $40 billion, things would start to change significantly.
AUDIENCE: You actually don't want a fulfillment in Massachusetts because then you pay shipping and taxes.
BEZOS: Yeah, you'd probably put it in New Hampshire.
AUDIENCE: How do you do that $25 deal? Do you get-- how does that work? No one else does it.
BEZOS: Well, that's why you should shop at amazon.com. I like this guy. Hey you're looking particularly handsome today.
It is-- the $25 Super Saver Shipping is incredibly expensive. So it has to be-- it's the most expensive price reduction we've ever made. And we're still hopeful, by the way, that we'll leave it in place permanently. We're going to make a decision at the end of the year about whether to make it permanent.
We were at $49. If we can't make $25 permanent, we'll go back to $49. But I'm very hopeful we can make the $25 level permanent. And the key there, is being very efficient. The economics can be made to work. It's just a question of, does it generate enough incremental lift in volume to justify the huge price reduction that it effectively is. Yes?
AUDIENCE: If you were a 2002 college graduate with your newly minted engineering degree, what are some of the fields you'd be most interested in working in?
BEZOS: Newly minted 2002 college student with an engineering degree, what are some of the fields that I would be most interested in working in? Well first of all, what I would say is-- the best advice I can give somebody, is do something that you think is interesting. And let the waves catch you.
Something that I see people do, which I don't think is a good idea, is they try to chase the current thing. I saw so much in 1997 and 1998, 1999, with respect to the internet. There were-- in '94, '95, '96, and '97, mostly the people who were-- the earlier you get, the more true this is-- but the people who are sort of looking at this space and started companies were genuinely interested in it. They thought it was really cool.
By the time you got to 1999, doctors were stopping-- you know, screw that doctor thing, I'm going to start a dot com company! So that really made no sense whatsoever. People were sort of abandoning the things they were genuinely interested and trying to catch a wave. Whenever you try to catch a wave, you're almost always too late.
So it's better to just-- I mean, if you're paddling after it. You basically have to wait in place and let it come to you. So just bring something that you're passionate about. I would definitely-- on the career advice side, the most important thing about your first job out of school is to pick a place where your learning per unit time is going to be very high.
You can optimize for a bunch of variables and it can get very confusing. You know, where's the location, what's the salary going to be? And all those things are worth considering and they're all important. But make sure you pull back after you've done your analysis and say, am I also picking the place that I believe my learning per unit time is going to be the fastest.
You learn much more from a best practices company than you do-- you learn no matter where you go, so if you go work for somebody and you're like, these people are all crazy, this is insane-- you'll learn a lot of negative lessons that you can use later and you can not do those things.
But that process of elimination method takes a long time. So it's better to try and find a place that has really high hiring standards. Some of the basics especially. You know, a place that's trying to recruit only the you know-- A's hire A's and B's hire C's and C's hire D's. So you got to hire only A's.
You need to pick a company that's doing those kinds of fundamental practices well, because the you'll learn so much that will set you up to do other things. So I would say that's probably the best career advice I can give.
HOST: We've got time for one more.
BEZOS: Time for one more question? Okay.
AUDIENCE: I was wondering about the international sites. I was actually buying stuff from different sites like Canada, US and Japan. I've noticed that these sites are actually really similar.
BEZOS: The site are similar. Yeah our international sites are similar.
AUDIENCE: I was wondering, how do you manage your international subsidiaries?
BEZOS: It's a very good question and I'm very pleased to note that you have found the sites to be similar. We insist-- all of our software development is done in Seattle. We don't allow any software development to be done-- we do have big physical operations and great marketing teams and so on outside of the US. But the software is so core and so expensive to do well, that you want to have exactly one set of source code that gets used all over the world, globally.
And many companies have made the mistake of allowing their different sets of source code in different countries to get out of sync and they end up with two completely different systems. Now they're getting no fixed cost leverage across those two systems. They've got two different teams, developing two different systems.
And then they decide one day, this is terrible and stupid, how did we let this happen? And they want to recombine those systems. And the systems by that point have diverged so much that recombining them is very difficult. So it's very important to have one set of source code around the world and that's-- thank you for noticing that.
AUDIENCE: [INAUDIBLE] distribution [INAUDIBLE].
BEZOS: But the software that runs the fulfillment centers is also the same all over the world. So the key is-- you know, what happens to software. Software is fascinating in a business because not only do you do all the things that you're trying to do, but some stuff-- what happens is business knowledge gets, over time, embedded in the software.
So little bits and pieces-- the software ends up knowing-- this is my opinion, again I admit to being sort of computer centric. But in my opinion, at the end of the day, the software knows more about your business than you do.
And so it's really important that you only have one such system. Because it's going to be complicated and it's going to be very interesting. It's going to have lots of knobs and the reason it has all those knobs, is because it knows a lot about your business. Thank you, you guys.