Brains, Minds, and Machines: The Marketplace for Intelligence

POGGIO: I'm especially proud of this panel. And thankful to the panelists for allowing us to put together a group of people that perhaps is even more interesting than each of its very interesting components.

The panel idea is to go back to the main theme of the old symposium, which reads as follows. Over the past 50 years, research in AI and machine learning has lead to the current development of a remarkably successful applications, such as Deep Blue, Google Search, Kinect, Watson, and Mobileye. While each of these systems performs at human level, or better in a narrow domain, none can be said to be intelligent.

So at the end of the symposium, we are going back to this theme. And I want to point out that often successes in science are measured in terms of publications or prizes given by other scientists. Now this is fine, but from the point of your machine learning, that smells of over-fitting. And it has dangers. So the ultimate test of success for science is really to have an impact over the long term, of course, on real life and for instance, on the marketplace. And so this panel is a marketplace for intelligence.

Now this close interplay between pure science and application is actually one of the genes of MIT. This is not a way our provost put it in a recent article, but he said, MIT pursues and values the most abstract curiosity-driven fundamental research and the most applied market-oriented innovations. And so, again, this panel is very much in this spirit. You have to show that the science you are doing can lead to things that do work in the real life.

Each of the panelists today represents one of the successful systems in the engineering of intelligence, which I listed in the theme of our symposium. They will cover state of the art achievements in our domain of intelligence, as I said, from Google Search to Watson to Kinect to Mobileye to medical devices and to future approaches. And each of the panelists also represent a company from well-known ones, such as Google, IBM, and Microsoft, to less known but successful ones like Medinol and Mobileye to really baby startups-- Navia and DeepMind. So behind the companies and products and ideas there are, of course, great people and I'll introduce each one of them in turn.

So first, Peter Norvig. You know, I find Google Search magic is better than any team of librarians I ever had when 30 years ago. Has changed so many things in the way we do things. Certainly, has changed the way I do research. I look for what has been published before what is known before I start really trying to solve a new problem.

Peter is director of research at Google. He's a fellow of the American Association of Artificial Intelligence. He has been faculty in various universities. He's the author of a very successful textbook, but also of a wonderful PowerPoint presentation, the Gettysburg PowerPoint presentation, which I recommend every one of you to look at.

David Ferrucci, let's see, I sent this email on February 14, 2011 to David. I said, David, sorry for the many emails. I'm watching the competition tonight-- this was the Jeopardy competition-- I'm watching the competition tonight in a very crowded MIT classroom. Could we invite Watson to come to our workshop? And David answered almost immediately. He said, not sure if Watson can join me, but I plan to attend. I hope it's still good enough. Watson steals the show. Thanks.

David is the lead researcher and principal investigator for the Watson Jeopardy Project. He's been research staff at IBM Watson Research Center since '95 where he led a team of 28 researchers and software engineers specializing in various areas critical to AI, software architecture, natural language processing, and so on.

And I want to say that even more than Watson, I was impressed by the organizational feat that David pulled out of leading and integrating in Watson so many different fields and people in his team.

Andrew Blake. Kinect is another magic application that just came out, is this box that used to play games with the Microsoft Xbox. It recognizes your gesture, the poses of your body. And the brain of Kinect, the machine learning technology which is inside, comes from the team of Andrew Blake who is the managing director of Microsoft Research Center in Cambridge.

Prior to joining Microsoft, Andrew was in academia for 18 years. Lately on the faculty at Oxford University. He has published several books. Was elected fellow among other things of the Royal Society a few years ago. He's an old friend, a real pioneer in computer vision. And among other contributions, a pioneer of the use of probabilities and machine learning in computer vision, which really transformed the field.

Amnon Shashua, I already introduced him yesterday, so I'll be brief. He holds the sax chair in computer science at the Hebrew University in Jerusalem. He received his PhD in computer science at MIT.

Working on competitional vision where he pioneered work on the most mathematical part of computer vision, which is really geometry, multiple view geometry bounding with this part of mathematics called algebraic geometry. I'm proud to say that Amnon was a student in the post-doc of mine. He's a great friend and a great colleague. And I said yesterday, I'll repeat today, that Mobileye is on top of my personal list of the successful applications of machine learning and computer vision in just the very recent past.

Demis Hassabis is a Wellcome Trust Research Fellow at the Gatsy Institute in London and a former well-known computer games designer. He was a child chess prodigy. He created the classical Theme Park for Electronic Arts, one of the best selling games at the time. And after selling a game company, he started a PhD in cognitive neuroscience at University College in London.

And he recently co-founded a new high tech startup to conduct research into general AI techniques. Among other things, he has won the International Mind Sports Olympiad five times, I think. He's a visiting scientist in my group, a wonderful dreamer of [INAUDIBLE] dreams.

Vikash. Vikash is a co-founder and CTO of Navia systems, which is a start up building probabilistic computing machines. He received his PhD at MIT in [INAUDIBLE], a BS, was a student of Josh Tennebaum. His PhD dissertation received MIT award for the best dissertation in computer science in 2009.

I still look at him as one of the bright students that make life a challenging fun at MIT for me. He's a great scientist, an entrepreneur, trying nothing less than changing the current paradigm in computation. You will hear from him how this will happen.

Kobi Richter. Kobi is one of these very rare people-- I know of two of them, including him-- who is very good at everything he does or did from flying fighter jets to all kinds of sports to art to science to business. He's a gifted multidisciplinary researcher in computer science and neuroscience with a PhD in broadly speaking neurophysiology from Tel Aviv.

He started recording from visual cortex while at MIT about at the time he arrived here in '81. Among other things, he was helping me negotiate with tough guy types used car dealers in Boston at the time. And at the same time, he was programming on the leased machines in my office in the AI lab and starting a company with Shimon Ullman in the field of computer vision called the Orbitech, which is still a world leader in the field of automated visual inspection machines.

And Kobi went on to develop new designs for coronary stents helping to create a dramatic transformation of therapeutics in cardiology over the last decade.

So with this, I'll ask Peter to give his presentation, around 10 minutes.

NORVIG: Thank you.

[APPLAUSE]

OK. Welcome all. Now one of the undercurrents of this conference has been let's compare the good old days to today and see how things are different, sort of this idea that maybe we missed something. And maybe things were great back then.

What I want to talk about, and maybe some of the others will talk about, is the great things that are going on right now. And yes, there is an opportunity to do more in the future to change what we're doing. And there's lots of good opportunities in the future. But there's also lots of great things going on recently and going on right now.

And since this panel's supposed to be about marketplace, I'll focus on the market impact of those great things, rather than the scientific impact. So let's try to estimate the economic impact of Google.

And by this, I'm exaggerating it, and using Google to stand for more than just Google itself and stand for the whole infrastructure of the web. Because what Google does is enable you to get to the web in the same sense that trucks have a big economic impact because they deliver all our goods around the country, but they don't make the goods themself. It's the same thing for Google. It's a transport system.

And Hal Varian took his attack at this at the Web 2.0 Conference recently. And you can look up his numbers. My numbers are going to be slightly different because I'm going to do the whole world, whereas he was just doing the US. But the main thing you should note is that there are really big error bars. And any of our guesses can be off by big factors.

Now the first thing to note is that we have advertisers. They pay us some money. They get some value back. And we can calculate that the value back to them is about seven times what they spend. So 88% of the economic value of Google is going back to them, rather than staying with us. And that works out to about $210 billion per year.

Then the next thing is, what about the values to the searchers rather than to the advertisers? And there's a few trillion interactions per year at the various Google properties. But it's hard to tell the economic value of that. If you watch a YouTube video of a kitten, how much is that worth to you? Well, you know, we can estimate it by saying how much are you willing to pay for a first run movie and divide by some constant. But maybe the economic impact is more negative rather than positive because you should have been doing something useful. So these are wild estimates.

But let's say something on the order of $7 to $73 billion for YouTube videos, depending on if you're willing to pay a penny or a dime to watch one. And then somewhere in the range of $37 to $370 billion for the searches at Google. And I'm only allowed to say that there's more than a billion searches per day at Google. We haven't updated that number in a while. But that's the official number I'm allowed to use.

And how much is a search worth? Maybe it's worth a dime. Maybe it's worth $1. If you have cancer and you discover the treatment that works for you on Google, that's worth a lot more than $1. If you use Google as a bookmark bar and you type in Yahoo and we send you to Yahoo, that's probably worth less than $0.10.

Okay. And then the value to websites. And there's a direct economic impact that there are publishers that show Google ads on their websites and we can measure that at $54 billion per year. But then there's the indirect of all the websites that get clicks and re-directions from Google for free. And maybe that's $250 billion. It's hard to tell.

And so the bottom line is that it works out to somewhere around 1% to 2% of the world GDP has been accelerated through this access to information. So that seems like a good economic impact.

What else is the impact of artificial intelligence, of machine learning, of statistics, of information technology? One is in experimentation. And Ronny Kohavi from Microsoft Research talks about this quick. Of these two, which one do you think would be better at getting you to click and make some money for MSN Real Estate? How many think it's a? Raise your right hand. How many think it's b? Raise your left hand. People aren't quite sure. Turns out a was 8.5% better.

Now look at these two. Which one is going to make you buy this product? Right hand if you think it's a. Left hand if you think it's b. And this time a was 60% better.

And you can play this game. And you don't have to do more than three or four before everybody's got at least one wrong. And the idea is that your intuition just is not reliable. And you've got to run these experiments. And having all this ability to do it online lets you do that.

And then having the data analytics lets you make sense of that in a larger sense. And there's lots of places now where data is available. And we can do more analytics.

There's all sorts of intelligent applications that have been enabled. So speech recognition, machine translation, self-driving cars, automated spam detection, genomic analysis, all these things are coming through new breakthroughs in machine learning, new capabilities for data analytics, and so on. Communications is important. And there's applications like question answering and dealing with e-mail and information overload and lots of opportunities there.

And then in education. So the US spends $120 billion a year. And the US is a quarter of the GDP. And maybe half of the education comes there and half the citizens pay for themselves. So it's a trillion dollars a year in education.

And currently, electronic learning is only $50 billion. And so I think there's a big gap there where it could be doing a lot more. MIT has 10,000 students, but the MIT open courseware website has 70 million students. And so what's the impact there? I don't think we have quite measured yet. The Khan Academy has 35,000 students a day. And the University of Phoenix has half a million students. That's a distance learning university.

Here's the Khan Academy. It has a bunch of videos that are online that are available for free. Students watch them and learn from that, either as a substitute or an augmentation to their own learning. And here's two examples-- one big and one small-- the MIT open courseware. Thousands of courses available where you can follow a whole course all the way through.

And then we're beginning to see applications as e-readers come on line, of things like the Amazon Kindle has this sharing of highlighted passages. So you can highlight a passage and you can choose to share it. And then you can see what other people think is important. And you can learn from that. And so according to Amazon, the most highlighted, the most important passage of all time is Jane Austen's it's a truth universally acknowledged quote.

Now, of course, in part she's helped because her book is out of copyright and so it's free to download, compared to the other ones that you actually have to buy. And she can't actually benefit from this feedback being dead. But for other people like me being a textbook author wouldn't it be great if I could get this feedback to say on this page this part worked and this part didn't work. And now I can go and fix it. And that's being enabled now.

And then in conclusion, let me just say that there's lots of big markets out there, some for which we've already had a huge economic impact on and others for which there's just money lying on the floor. And if you pick it up, you get to keep some. And like Google does, you get to share most of it with the world to make the world a better place. So go out there and pick it up.

[APPLAUSE]

POGGIO: David.

FERRUCCI: So thank you. Thank you very much. It's really an honor to be able to get a chance to speak to you today. I want to make a couple of points. I don't have those kinds of numbers that you've just seen about the economic impact of Watson. Of course, we're very excited about it. We have a tremendous interest from industry, from lots of customers coming to IBM. We've sort of captured the imagination of the broader audience. And all of this is having very positive potential from a business perspective with IBM.

I want to talk about three main points, which is during the course of this project we learned a lot. And one of the things we learned about was of the opaque perceptions of AI and computers that we found as we were communicating to the general audience about what Watson does and how it does it. And I wanted to share a little bit of that with you.

I also wanted to share a little bit of how we focused on engineering and engineering for science. And what some of the commitments we made to succeed with building Watson. And ultimately, you know the importance of the right interaction can really change the business. And I put interaction in quotes there because it's really about how you interact with computers and how you communicate with the user and with the general audience and how powerful that can be in impacting business and the understanding of how to apply technology.

I got an opportunity to see a couple of the panels before. And I learned two things. I think one was that brains are really sexy. And warning, there's not a single picture, I couldn't resist, not a single picture of a brain in this presentation. But I think the other thing I learned is that in this few just 10 minutes I have that these audiences can absorb someone speaking really, really quickly. So that's what I'm going to do.

So you know, first off, these bipolar impressions we got. I actually did a radio interview where someone asked me pretty much exactly what was there. Okay, so you put the questions and the answers in, and then when the clue comes up, Watson speaks the answer. Right? And I was like, no. We don't know what the questions or the answers are ahead of time. And the host said, well then how do you do it? I said exactly. Right? So true story.

But you have very different impressions. On the other hand, we would get people who would see some of this and say, oh my god, you know, it's frightening. It's Skynet. When's Watson going to enslave us? Right? So you had this complete opposing things. Even within IBM, people looked at this initially and thought, gee, it's too easy. In fact, some people thought it was name that tune.

Other people, on the other end of the spectrum, thought this is too hard. You know, this is pure folly. And you're just going to fall on your face and embarrass us.

So it was interesting to me that you had these bipolar impressions about what was going on. And this was really not just a study of the audience in fun. This really affected to what extent IBM, my company, was willing to make an investment in this, what I considered, very much an AI problem, an AI challenge. And for me, it was irresistible.

What I ended up with is trying to communicate more about what computers find easy and what they find hard. And looking for ways to communicate that. I had a unique opportunity to talk to many, many former jeopardy contestants who came and sparred against the computer. And I had this challenge.

It was amazing that they were more interested in how the computer did the strategy or how it decided to bet, because this is what they struggled with. Their unconscious of how. Or I'm going out on a limb here, but their unconscious often of how they actually answer the questions. It happens in a blink of an eye for them what's going on.

And what they are conscious of is how the computed a bet or how they did the math or how they did the strategy. And of course, that was largely the easy part. And I don't want to undersell some of the cool things we did there, but largely the easy part with regard to the technology and the effort. But something they really struggled with.

So one of the things I did was talk about what's easy and what's hard for a computer. And people can look at you. You had a long time now to look at that. Anybody know the answer? No. So I bet you weren't even sure if it was less than or greater than one.

So this is easy for a computer, looking through tables of information-- here's a pseudo SQL query-- is easy for a computer. And this helped communicate that. This is how computers work. And it's a paradigm we're all sort of very used to.

And even people who are not experts in computers use spreadsheets, they use databases, and they know if you put your serial number, you could look up your address. So this is kind of what they expect when they see a computer just getting a question and answering it. They're expecting you're just looking it up in a database.

And what I found was that-- at the speed of light here-- what I found was what helps communicate the challenge that we have with natural language processing and how to contrast that with database tables was this notion of what goes on with here you have this table, you wonder where somebody was born, and you could look it up in structured information.

But what if you had the same question and you had no idea where to go or how to answer it, how to interpret it? And you had to read where was Einstein born and infer that from that text? One day from among the city views of all [INAUDIBLE] Albert Einstein is remembered as Einstein's birthplace. You know, you have some cues there, but now you have to understand there are people, there are places, there are relationships.

Here's another example. X ran this. Now sure if we knew that question was answered by that table we can look it up. But instead if we had if leadership is an art, then surely Jack Welch has proved himself a master painter at his tenure at GE. We have a slightly different problem here. So with some confidence, we may deduce that Jack Welch was a painter at GE. But that, of course, is not what was intended by the question, nor really the answer.

And this is how people sort of scratch their heads and go, oh, I'm starting to understand really what the challenge is. And I think what was neat about Jeopardy from a scientific perspective was it helped us tackle some key points-- the broad open domain. So you can't anticipate what the questions were going to be about, or exactly how they were going to be phrased. Complex language.

And you have to have a very, very high precision. But moreover, the computer had to know what it knew in the sense that it had to know what was right. Because every time you say you know an answer, and you get in there and you buzz first, if you get that question wrong, you lose the dollar value associated with that question. So there was risk in answering with the wrong answer. So it had to compute, essentially, a probability, an accurate probability, as accurately as we can get, that you got the answer correctly. You got it correct. So you could take that risk and buzz in. And you have to do this very, very quickly.

And this was really the foundation of what the engineering feat was. And it involved starting with really, really crisp metrics. One of most important metrics we had was represented by this winner's cloud. And I can't tell you how important having a crisp metric was for both the science and the engineering of the problem.

So we didn't start off with this general notion we're going to build this big AI. It's going to be really smart. And there's going to be Maxwell equations for doing question answering. We sat there, and we were very, very practical.

In fact, someone came to me and said, you know, I'm a cognitive psychologist. And I really, really want to be in your project. And as much as I love to philosophize about cognition, I had to get really practical really quickly. Because I had to meet with the senior VP quarterly who said to me, Dave, you're mucking with the IBM brand. Are we going to win?

So this was very much an engineering feat. That baseline was where we started with a QA system. And we had to drive that conference curve up so it was basically splitting right through that or cutting right across that winner's cloud. Because that's where winning players were performing. So if they were confident enough and fast enough to get a chance to answer 50% of the questions on a board, they were doing between 85% and 95% precision.

I'm going skip some of these slides because I don't have the time for it. But I want to get to a few guiding principles. And this is kind of for me were a lot of the AI theory sat for me. Right?

So we made a couple of key commitments. Large crafted semantic models will fail. Too slow, too narrow, too brittle, too biased. So we need to acquire and analyze information from as is bull sources. So we consumed things like Wikipedia and many other encyclopedias, dictionaries, thesauri, entire plays, books from Gutenberg press, the Bible, Shakespeare, and you name it.

We had the expectation that there was no one algorithm that was going to be able to do question answering, but that intelligence would come from many, many diverse algorithms. And that each were going to react to different weaknesses in the system based on doing error analysis and repeating and improving the core NLP that was in there to attack each problem we would start to see and generalize from.

And the relative impact of an integrated metric must be automatically learned. In other words, there was no presumption that if you got your parser from 95% to 98%, or if you got your information extraction from 75% F measure to 80%, that this was necessarily going to impact the end to end metric of the question answering system.

So instead, what we had to do was do this empirically. Run many, many, many experiments to understand how these various algorithms and features interacted to effect the end to end metric.

Another assumption was that while we were going to put many, many algorithms in there, and let different hypotheses compute throughout the system, that in order to deliver the latency, which we needed, which was to answer a question come up with an answer and an accurate probability in three seconds, we are going to have to scale out. We were going to have to have an algorithm that was embarrassingly parallel, which is exactly what we did. And this is a high level picture of Watson's brain, if you will, even though I promised I wouldn't show a picture of a brain. I meant a human brain.

So it takes questions and categories. It comes up with many different interpretations of that question based on doing deep language parsing and semantic analysis. It generates many searches from those hundreds of possible answers. Each of those become the root of a tree where it would collect evidence to support or refute those answers.

And from there, it would apply many different scores, hundreds of different scores that would analyze that content from different directions. It would weigh those various features based on machine learning models. Have an end result of a ranked list of answers with all of them having a probability that they were correct.

And if that probability was over the threshold, then Watson would want to buzz in and answer. And I'm going to go over it just a little bit. We populated that architecture with hundreds of algorithms. We had a sort of a goal-oriented metrics. We did regular integration testing. We needed the hardware necessary to support that engineering, which gave us those results we needed to push the science.

It took two hours to answer a single question on a single CPU. We ended up scaling that out over 3,000 cores to be able to answer fast enough.

In the end, we were able to drive that performance from that baseline all the way up to where it was cutting across that winner's cloud. And this was a big test on a blind data-- 200 games worth of blind data. Does it guarantee a win? Absolutely not, but you could see you're competitive there with the grand champions across that curve.

And the last thing I want to say is-- actually, this is not last thing. I have two more things I want to say. One is that the integrated metric, there was a system level end to end metric that we focused maniacally on with the presumption that winning on Jeopardy was something that was going to advance the size of NLP. We still had to focus on that end to end system metric.

When we were done with that, we went back and looked at some of our core component NLP. We looked at how it evolved with regard to independent component level metrics, the kinds of things you typically publish on. And we're going to write all these papers. But we were leaders in all these areas from parsing to disambiguation to relationship attraction to passage mashing and passage scoring. In every one of them we were leaders based on those metrics. So that was a very, very satisfying thing to discover. If you choose that problem right and you have metrics and you push that, you can get there. So it was very satisfying.

The last thing I want to talk about is evidence profiles, and then I'll stop. In Watson, we had many, many features-- many, many algorithms producing many, many features. Just to get a handle on what was going on, we had to build tools to understand how those algorithms were interacting and impacting the correctness of an answer.

Eventually, we got to what we called an evidence profile where you'd be able to group out of hundreds of features, you could group them into semantic categories. And them based on the machine model waves, you would be able to see how they contributed to a right and wrong answer. This allowed us to do error analysis, to understand where the system was failing. This was a very, very powerful capability ultimately.

And what to do to solve problems. There's a number of things going on here. But I want to point out that one of the issues in getting this question wrong here, it says, you'll find Bethel College and Seminary in this holy Minnesota city. Actually, Bethel College and Seminary in St. Paul, Minnesota and South Bend, Indiana. We didn't weigh location information high enough.

But there was also another path to answering this, which is through the pun, the association between holy and St. Paul, the right answer. Actually independently one person came up with some ideas on how to detect puns in language. Was able to integrate that into the system, train it, and that was having impact on the end to end system. And we were getting more pun questions right. We did this very, very rapidly because of that architecture we depended on.

My last point, the notion of an evidence profile in the communication of confidence ended up becoming very, very important. When you watch the game, where all you saw was the computer answering when it knew an answer and getting it right most often right, you thought, gee, it must be just looking it up in a database. What's really going on here?

When you saw just a glimmer of what was happening inside the machine, when you saw those confidence values and that histogram, the general audience starts to scratch their head and say, wait, there's something else going on here. You mean a computer can maybe be wrong? You mean it's not sure. You mean it's thinking. I mean we think of it as computing probabilities based on analyzing evidence.

But this got people to think, wow, this is really interesting. And this inspired both the general public to think about what is AI really all about. And it also inspired our customers, to think, you know what, this notion of dealing with uncertainty and dealing with language effectively helps me address a problem I didn't think of that way before. And we got lots of interest, both from the general public and from our customers. Thank you.

[APPLAUSE]

POGGIO: Andrew and Microsoft.

BLAKE: Thanks very much for inviting me, Tommy. It's been an amazing workshop the last two days. I guess one thing you didn't mention from my CV is that I studied here for a year. And I didn't get a degree at the end. So I guess that makes me an MIT dropout, which I guess must be something to be proud of.

So I just wanted to talk a little bit about technologies for natural interaction and some of the progress that I think is happening there. You know there are many contexts in which we want to do natural interaction. Some [INAUDIBLE] it is.

So the one that is sort of the source of my talk, if you like, is gaming. And that's kind of obvious. But people are thinking about all kinds of other things, like in medicine, for example, bringing some kind of intelligence, albeit superficial, to bear in analyzing what's going on in the human body.

And here's another investigation of natural interaction. This time in the operating theater that it's not really practical for a surgeon to get up and start poking at a keyboard and manipulating a mouse. If they do that, they'll have to go and scrub up again afterwards. So it would be great to have some more flexible non-contact kind of interaction with the environment.

Cars, well, Amnon is going to talk all about cars. More and more the high end ones, at any rate, at the moment are coming up with head up displays. And so there's a lot of opportunity there for a more natural interaction with the information that's displayed on the screen, for example, pointing navigation information, which is maybe projected onto the world.

Here's a rather beautiful robot that the Anybots Corporation has built to be your proxy at a meeting. It will turn up to the meeting and people will talk to it. And I don't know if that's going to catch on. It's a fascinating idea.

What else? So of course there's Minority Report. I don't know if we really believe in that, that that's what we actually want to have as the interface for interacting with information. But again, a fascinating idea.

So you know the technologies for interaction are coming on. We've had history of necessarily clunky things getting more and more flexible. Now touch and multi-touch and then the new generation that I want to talk about, you could call it no touch or action at a distance.

So the Kinect camera that Tommy referred to came out earlier this year. And yes, that's what my team has been working on putting machine learning into it. And here is the team. I'm going to tell you just a little bit first about what we did. Oh, but before that, in case you didn't play with it yourselves, here are the kind of things you can do. You play games with your buddies. I'm not a gamer myself by any stretch of the imagination, but I must admit I have found some of these Kinect games are quite compelling just because they're social and they're easy to interact with. They're not too kind of head banging.

So what do we have to do to make the Kinect camera work? Well it's sensing the pose of the human body. And so the task is to receive the signal that the body generates and reduce that to a set of joints, rather like motion capture systems that they use in movies a lot. You know, in Titanic when the guys all drown in the sea. Actually what they do is they put the guys on a slide and they capture their motion as they slide into a pool. And then transfer that into the scene.

So what we needed was to build something like that, but without having to attach all the instrumentation to the body of the actor, because, of course, that doesn't make sense in the interaction setting. So we take the body in in the form of data. And what we have to put out is the skeleton inferred on the right there. And represented mathematically in terms of positions and angles and so on.

This is a hard problem because the human body is amazingly pliable and variable. Even your body can do many things. And then, of course, there are many different kinds of bodies varying in size with unfortunate accidents of appearance and in very complex settings and so on. So we needed to make significant inroads into these complexities.

And it's a kind of commonplace of AI one of the lessons that AI has established is that trying to program rules explicitly to handle situations of this complexity doesn't really work. So we realized we had to do this somehow by machine learning. What we thought we would start with was principles based on the work of Dariu Gavrila. He this work in the Daimler Benz research lab. He also was anticipating bringing a vision to cars. And he had a fascinating system of hierarchical templates of the human form capturing the position of the human form in a cluttered scene.

We thought that was not quite going to be enough because of the need to capture all of the articulations of the limbs, which you don't really have to detect pedestrians. So we were thinking in terms of something involving templates for parts of the body. But then you'd have something to do afterwards to harmonize those templates if they had all detected their parts independently.

Meanwhile, Jamie Shorten on my team had been working on I guess we can call it an AI problem. At least there are an awful lot of people working on this at the moment, which is let's get back to object recognition now with what we've learned in the last 15 or 20 years about low level vision and see if we do any better this time around.

And well, we don't have human level object recognition yet, but amazing amounts of progress have been made recognizing up to 100 objects at once in a scene. So Jamie wanted to use this kind of technology to try to solve the problem of detecting the human body in depth maps because the Kinect sensor is a 3D sensor.

And my contribution at this point was quite crucial. I told him that it would never work. And he, nonetheless, went on playing with it. And I said, well, that's fine. Try it. Why not? But we better pursue both tracks because yours is never going to work.

And then after a month, his was working surprisingly well. I mean, it's still not working properly. But I said, well, I think we should still hedge our bets and do the two approaches. But fine, you play with that if you like. Your Friday evening project, except every day was Friday.

And after a couple of months, it was really working amazingly well to an extent that I never thought it would. And so we kind of gave up on the harmonizing the templates idea and went with the recognition.

So this is what we ended up doing. The depth amount that you receive from the sensor is on the left here with close stuff coded as bright and far stuff coded as dark. And we ended up with an output which is like a kind of noisy clown suit. Actually, the input I'll show you soon in at least the training data is a kind of perfect clown suit. But the clown suit here is colored according to the body parts.

And then as the guy moves, you see the clown suit is moving pretty much with the guy. And this is what convinced me finally, okay, the clown suit doesn't fall off when you move around. So the labels really are sticking to the body.

Here we have many different poses, all labeled with the same clown suit. Of course, there are 31 parts of the body labeled here and various things happen, which I don't have time to tell you about to generate hypotheses, which eventually boil down to tracking a number of joints in the body. So that was the story vastly accelerated. And that kind of capability takes its place amongst I think a number of capabilities that are coming out that will change the nature of interaction with machines.

And of course, our aim is to make the computers disappear in our interactions with machines that we find around the world. So we don't want to think of our cars as computers or assemblies of computers. We want to think of them as cars. And I think this is true for all of the machines with embedded computing that we use.

So of course, speech technology has advanced incredibly over the last 15 years. And I think that is one of the huge areas of progress. I would like to disagree with Patrick I guess what you were saying a couple of days ago. You know this is a field which I think has been transformed out of all recognition. And you know, hey, let's call it artificial intelligence. I don't know if it exactly is or isn't, but let's claim it. And that has been a huge piece of progress.

Itrackers, incidentally, are something which all of a sudden seem to have come of age. You can get them embedded in computers now, other kinds of sensing. I've just got to show you one or two things. A couple of things from MIT. This is something I saw from Ross Picard from the Media Lab. It's amazing that you can sit in front of your laptop now, and without anything special attached to the laptop, it will tell you how fast your heart is beating. That is truly astonishing when you first see that demo.

And another thing which migrated from one Cambridge to the other, Rana el Kaliouby, who was in Cambridge University and now is here in the Media Lab, has made amazing progress in the interpretation of emotions. For a long time, computer vision researchers were grinding on about is it the six basic emotions and recognizing those. In turns out not to be very difficult.

Once you get into kind of complex emotions, then the emotions have trajectories in time. And everything gets more complex. And Rana's work is so beautiful because it exploits autism databases that were constructed in Cambridge Simon Baron Cohen for the diagnosis of autism and turns them into oracles for what emotions are as reflected in the face.

Something else we're doing in Cambridge is trying to make a common place out of the business of gesture recognition. The guy is controlling a DVD player, by the way, using his hands. And of course, you could build this into the Xbox with Kinect.

I'll just advance it so it stops making that noise. What's special about this is that we never actually built any kind of a recognizer for the gestures. All that happened was we recorded a DVD with examples of different gestures on it. Fed that into a machine learning algorithm. Out of that dropped out a recognizing module. So you could imagine a designer, somebody who doesn't want to think about computer science, building an interface to their machine using this capability.

Well okay, I'm over my limit. So I'm just going to skip to my-- how about this slide? So just one other thing I wanted to mention is just the whole idea that perception, which is what we're trying to build into these machines that give us flexible experiences with devices, is enhanced or relies on having probabilistic reasoning built into the machines.

And a kind of lingua Franca for probabilistic reasoning, much of this theory was developed here in MIT and elsewhere, is the idea of the probabilistic graphical model. Here, for example, is a graphical model that encodes asmer, the different causes and effects and possible intermediate concepts. Here is a probabilistic graphical model that encodes game playing and the way in which your skill will affect your performance and how that plays out with multiple teams.

And just to finish, I just wanted to say that one of the things we're working on in the lab is trying to embody this in a package which could be handed out to developers. And developers are trying it as we speak, it's on the web for people to download and play with. We call it infer,net. And it encapsulating all of that graphical model language, if you like, in a way which we hope ultimately will become a commonplace in software development. We're not really there yet.

So at the moment, it's still a good idea if you have a PhD in machine learning to program with this stuff. But we hope that in the future this will become commonplace.

So just to conclude, a lot is happening I think which gives us reason for hope about having more natural interactions with machines in the future, new forms of sensing and sensory processing, and that crucial ability to program with probabilities. Thanks.

[APPLAUSE]

POGGIO: Amnon.

SHASHUA: So the title is a bit presumptuous. Given the gap between our engineering successes and the wide spectrum of human perception abilities, the gap is staggering. So the title itself doesn't have any meaning. It's not well-defined unless we narrow the domain. And this is the secondary title that I'm going to talk about the domain of making the car see.

And I'll be modest. I'll be talking about technologies that I think can be implemented within five years, 5 to 10 years. Where this field of making cars see has the potential to reach within five years and what are those technologies that we'll need to develop.

So just a slide of recap. Today's technologies centered around a camera looking forward front facing doing about five different functionalities, lane departure warning or lane analysis, detecting cars, performing forward collision warning and braking on cars, detecting pedestrians, and doing the same thing on pedestrians, detecting and reading traffic signs about 60 to 80 different traffic signs. And also doing certain lighting functions. At night, be able to automatically determine whether the high beam should be on or off.