The World Beyond

Transcription

And a very good afternoon and welcome back. For the people that just joined, I'm David van der Leer, a co-chair with Charles Kaden for this conference. And we're starting on our run to the end of the afternoon. It will be an interesting afternoon, I feel. And I would like to look back briefly at what Gerald said early this morning. He was talking about all of these competitions, even mentioned Drupal. And there were few competition that Gerald didn't mention. So for instance in China, they have this crazy competition in which people gets covered in bees, like hundreds of thousands of bees on one person. Don't know why one would. And now I'm not sure what it's for. And then in Holland, we have this art competition in which we still eat cake from strings. And you can now eat it with your hands behind your back. What I'm trying to get to is that competitions don't only take place within the design field. And this afternoon we're starting with a panel titled The World Beyond, and we have three fantastic speakers. I think of some, and I know of most, who will be talking about what is happening outside of the design field in these competitions. First up, we have Jenn Gustetic, who's working at the White House Office of Science and Technology Policy. Jenn is slowly getting well known to Van Allen. So we're happy to see you again. Then our second speaker is Nidhi Chaudhary. Almost good? Yes. Very good. We practiced over lunch. [inaudible] at HeroX. I'm happy to have you here today. And then our last speaker was certainly not least, is Bill Aulet, for the Martin Trust Center for MIT Entrepreneurship. So great to have all of you here. And let's kick it off with Jenn. I won't do introductions in between. So Needy, you can go straight on after that. Hello. And I have the coveted right after lunch spot, which means I will try to vary the pace of my talking, and the loudness and quietness to keep you paying attention. So good afternoon. And I am happy to be here and to share with you guys some of the prizes and challenges experience outside of the design world that have been occurring for the last several hundred years as well. Many of you may not know that Charles Lindbergh actually flew across the Atlantic in response to a prize competition. It was called the Orteig Prize. Back in the early 1900s, there was a $25,000 prize for the first nonstop flight from New York to Paris. A lot of people died trying to actually accomplish this prize. We say that in the prize space, that there are four reasons why people participate in prizes and challenges. The 4G's. Gold, guts, goods, and glory. In this case, it was very much guts and glory that he was competing in this particular prize competition. But the reason that the Orteig family offered up this prize was also to break people's conceptions about-- or misconceptions about whether or not regular Atlantic flight was a true transport option for folks. At that time, this was not obviously-- commercial flight was not something that was a regular way of people transporting themselves around the world. And so this was really kind of a paradigm-breaking activity. A demonstration to show that it was possible to safely do this. To try to unlock, potentially, a whole new industry which became largely commercial flight as we see it today. This was mimicked by a group called the XPRIZE Foundation in the early 2000s with another prize called the Ansari XPRIZE. That was a $10 million prize for the first group to create a suborbital launch transportation vehicle that would be able to launch, take two trips around the earth, land, and do it again in two weeks with the same vehicle. So a reusable, suborbital, kind of commercial space tourism type of vehicle. And in many ways that unlocked the companies now that we see starting at the very beginnings of a commercial space tourism industry. Folks like Virgin Galactic, and others that are playing in this space. So point here is that we've seen prizes around for a long, long time. Not just in the design competition space, but in kind of the technology, development, demonstration, and scientific discovery space as well. Though there has been a recent renaissance. You'll see on the left hand side of the slide some data from a McKinsey study that was done in 2009 that showed a significant increase in the amount of price funds that had been made available over a four decade period from the early '70s until when that report was published. And so there is certainly a trend being recognized at the beginning of the Obama administration. And the Obama administration, recognizing the power that prizes could have in transforming particularly difficult and hard problem spaces, really, really supported the use, more use of these tools within the federal government context. So this has been incorporated in many government-wide strategies like the Strategy for American Innovation. But also Congress has also supported this notion, authorizing the use of prize competitions for every single federal government agency in 2010 through the passage of the America Competes Act, which gives every agency the authority to conduct prizes. So since then, since 2010, the federal government has conducted over 400 prize competitions. It's a lot. With almost every single federal agency having conducted at least one. 72 agencies. You can go to challenge.gov which is the one stop shop for all federal-sponsored prizes and challenges. Almost $100 million in US government money has been spent this way as well. So it's a place where real money is being spent. The $64 million is from the 2010 to 2014 set of prizes. 319 of them. But since then, 80 more have been conducted since this particular study that's being referenced on this slide. And prizes do a lot of different things. So design challenges and prizes-- we heard a lot about-- earlier today about how kind of design challenges are defined. What the elements of a design challenge are. And there's a lot of similarities to prizes. We talk about them in the government context. So you set a problem statement, you have competitors, you set rules and judging criteria, and then you either have objective or subjective judging. I'll give you some examples later of some objective judging that we do in the prize competition space that's not as subjective as some of the design challenges that we've talked about today. And then you award winner-- a winner, or winners for prize competitions. But we design prizes to do a lot of different types of things. And they produce a lot of different types of products. McKinsey describes six different types of prizes kind of archetypes that they'd seen up until the 2009 period of what prizes could look like. So these are prizes that range from those are supposed to stimulate new markets like the Ansari XPRIZE, to prizes that are more like the Nobel Prize. Exemplar prizes. So these recognize good work that has already been done. Not necessarily incentives new work, but are things that you are recognizing good work that has already been done. Oftentimes you can actually mix some of these types together too. So you might do a exemplar prize first to recognize the best and brightest in a field, if you don't already have that list kind of fully accessible. To then leverage that list of folks as a potential follow on invite list to a follow on incentive prize, or potentially as your set of judges, or your amplifiers for another type of incentive prize competition. So you can couple these together a lot as well. Now I'm going to run through a few examples to kind of give you a sense of what these outcomes can look like on a variety of different types of problems. And so many of you probably watch the news. And in the fall, it was dominated by Ebola. There was a lot of fear about this topic here in the US, but also the reality of a lot of individuals in West Africa dying and health care workers that were not being given the tools that were helping them to not only be effective in their treatment of patients, but also comfortable and safe. And the Personal Protective Equipment, PPE that these folks use in infectious disease treatment hasn't been innovated on for decades. And in the sub-Saharan Africa context in treating Ebola, at the end of their rainy and hot season, imagine being covered head to toe in like, a plastic garbage bag, and not wanting to die of heat execution in 40 minutes being within that environment, let alone having your helmet fog up and being unable to actually treat optimally the patients that you're going in to see. Recognizing the way that health care workers were equipped and the response to the Ebola crisis was not as good as it should be given all of the innovations that we have in sporting equipment and other types of fields, why can't we apply some of those same types of procedures, and materials, and methods to something like this. And so USAID, the US Agency for International Development and the White House launched a grand challenge in the fall of last year to try to identify more innovative approaches to protecting health care workers and the response to Ebola. Now what's interesting about this particular challenge, is there was a three-pronged approach to actually getting these from idea to implementation. The first was opening up the ability for folks to inject design insights into the process that we may not have sourced otherwise. So we did a large open IDEO, which is one online platform. IDEO is a design firm that many of you have probably heard of. They do design, not so much in the architectural space, but the designing a better mouse, or designing a better toothbrush, those types of ways of thinking of design. But they have an online platform called OpenIDEO that is all around kind of crowd sourcing the design thinking method to particular problem areas. And so we posted an OpenIDEO challenge that was seeking design ideas, insights, empathy, and different approaches from other fields towards this particular problem area. It was open for about a month. Got nearly 1,500 ideas in through that process. And coupled with that in-person workshop. So in-person design-a-thons, where folks kind of from around the country convene their own design jams, where they actually brought the current personal protective equipment that are sold by folks like DuPont and others to the event and hacked the physical PPE. So there was one team actually from John Hopkins, John Hopkins put together a hackathon up in New York that a wedding dress designer decided to come, because she saw the announcement in her local paper that this event was happening. She wanted to try to do something to help. And she ended up being a member of a team that she was the critical member of the team that actually built the new suit, sewing it. Being someone who was really familiar with working with materials and putting together kind of complex suits. And that team ended up being one of the winners of a follow on $2 million funding opportunity that USAID offered to try to help take the best ideas and get them the ability to be tested in a government facility. So many of these ideas, if they weren't actually tested and certified, these would never see the light of day actually in the field, because they only certify things that are safe for use in these environments. And so in this particular case, I think it's interesting because we used three different tactics. An online ideation platform, in-person hackathons, as well as a follow on funding call for the best ideas to actually be fed into testing and further development. And this all happen in three months, which is like beyond fast for the federal government. Another example that's close my heart-- I am actually from NASA I'm detailed to the White House for a year and a half, but NASA does a lot of really cool prize competitions. And we've been doing them since the mid 2000's. This is an example of a prize competition that was run in 2008. So it might seem a bit old, that's seven years ago, but oftentimes that's how long it takes actually for companies that come out of prize competitions to find ways to get their product to market. So a lot of the best examples that we have, from really transformative technology development and demonstration prize competitions tend to be five or more years old. This particular one was a very objective prize competition. So the previous one, they were paper proposals that people were evaluating against a certain number of factors. With this one, you actually had two or more companies bring a lunar lander. So that is a lunar landing technology that was physically brought to a demonstration facility over the summer of 2000 and-- this one was 2009, that had to be tested head to head against another team to see how well it objectively performed taking off at a height of 50 meters, translating 100 meters horizontally to another landing pad, and landing to like centimeters of precision. They were ultimately judged on who made it closely-- most close to the target, on a matter of centimeters. So very objective. I mean, if you're watching that, you either did it or you didn't. Your technology was able to meet the challenge, or it wasn't able to meet the challenge. And in this case the winning team got $100,000, but the second place team also got just shy of $500,000. And so with most of NASA challenges, their are first, second, third place opportunities for cash. And oftentimes it is the second place winners that end up being the most commercially successful, which is also interesting. Regardless of what place it is, they use the distinction as being a NASA prize winner, regardless of whether you're first, second, third, fourth, fifth to help you in kind of your commercialization endeavors as well. Now both of the companies that won this prize, first and second place, have contracts with NASA. But it was not as a result of this prize that they got the contract with NASA. It was follow on work that this prize made them a reputable aerospace company, that they were not before participating in this prize. Masten Space Systems, that's listed at the bottom, says that when they were a participant in this prize competition, the prize competition was their only client. They didn't have another client. But now they are a legitimate aerospace industry player, on a small business level. But now they have multiple customers in the space industry, and they needed this challenge in order to legitimize their work, largely. This one very quickly. Prizes can also be used as data collection activities. So very quickly on this one, community colleges and education in the country is another issue that the White House cares a lot about. And one of the difficulties in this particular area is it's hard to know what to do from a policy standpoint, if you don't have the data about what's working and what's not working. And so with the Aspen Community College Excellence Prize, this was largely a data collection activity in sheep's clothing. What they did is they identified what they thought might be the top 30 metrics for what would make one community college better than another, from publicly available data. And they gathered from that, they put together a list of Top 100 Community Colleges. What they thought might be the best 100 the country. And invited those 100 to apply to be distinguished as the Best Community College in the Country, as conveyed upon them by the Aspen Institute. And out of those hundred that they invited in the initial year, about 70 actually applied. And in order to apply, you had to submit this nice 15 page document that had all the sorts of data about your university that had been designed by an evaluation firm trying to gather the types of information that they would need to understand what makes objectively this practice better than another practice, and this college better than another college. And they're now in their third year of doing this competition. So every year now, they iterate upon the survey, and are collecting more and more sophisticated data about what we know about the best performing colleges in the country in order to inform better policy making in this area. Last one, also one of my favorites, because it's NASA. So you may know that there are asteroids out there. You may know that they also killed off the dinosaurs, so they would be a bad day for us if a big one was ever coming towards us. We don't want that to happen. Again at NASA, because it is one of the few natural disaster risks that we could do something about. If you identify them early enough, you can move them. And by the time they're supposed to hit, we're in different locations. So this is actually-- we can't do much about earthquakes, we can't do much about hurricanes in terms of preventing them, but we can prevent this one. This bad day. So NASA has launched an asteroid grand challenge to find all asteroid threats to human populations and know what to do about them, recognizing that there's a particular part of asteroid detection that our scientists might not be the best at, which is optimizing algorithms. Our scientists are world class at characterizing asteroids once we have data about them, but not necessarily at parsing through thousands and thousands and thousands of pieces of data in order to understand where all the potential asteroid threats might be out there using data from our telescopes. And so we did a challenge on topcoder, which is another kind of OpenIDEO-like online community. $74,000 in prizes later, we were able to develop with a coating community and a data science community. So this is not an astrophysics or planetary science community. It's out of discipline. That's really important that we were engaging out of discipline skills to incorporate those particular things that they're better at into the system at NASA to make the whole system richer by engaging disciplines that we don't traditionally engage, and got an algorithm that is 15% better at detecting asteroids in the main belt, between Mars and Jupiter, which is huge. And none of the scientists thought that was going to be possible. Finally, I will say the government can play a lot of roles in prizes. We can host them and run them, which we do a lot of. We can coordinate them, meaning we can lead the process of identifying problems that would be useful to solve through this process, or we can contribute. So we've done a lot of things in the past where we've provided test facilities. Where like the government has the only test facility in the country that you can actually spill a bunch of oil in and test the oil cleanup technologies. So for the Wendy Schmidt Oil Cleanup X Challenge, that was run by the XPRIZE Foundation, the government provided the testing facilities so you could actually do the objective head to head competition for oil cleanup technologies. And finally, I'll leave you with this chart, which shows the prizes and challenges, crowdsourcing in citizen science are one of a number of different options that the public sector and the private sector have in order to solve problems. And they're not right for every type of problem, by no means. I'm the first one to say that prizes are not the right tool for every problem, but they can be transformative for the right kinds of problems. The other thing that I will say is that if you've seen one prize you've seen one prize. They're all very, very different, and thus prize designers and the active prize design is more like a craft than it is a science. It's more like an art than it is a science. We're still learning a lot about what makes prize design best in some situations-- better in some situations than others. So with that, I look forward to questions on the panel discussion at the end. Turn it over to my colleague. Hi everyone. It's actually works out really nicely. Jenn covered most of my slides, so I can cut my presentation to five minutes maybe. She actually did cover quite a bit, so it's really wonderful to see the synergies between obviously what government is doing and what we're also doing at HeroX. HeroX is a spin off of XPRIZE which has been mentioned a couple of times already. So I'll start with the story again that Jenn had mentioned, which is the $10 million Ansari XPRIZE. And so for those of you who don't know, Peter Diamandis is a man of grand visions. He started XPRIZE Foundation about 14 years ago. But he's a man of a rather small stature. He's 5' 4" and he can't-- he wanted to go to space and NASA said, no you're not tall enough. So you're not allowed to go. And he thought that was rather unfair. So he created the XPRIZE to send someone to space. And then the Ansari family stepped in and offered $10 million to fund the prize. As you can see, it was launched in 1994 and then award in 2004. And again as Jenn mentioned, this sort of revolutionized the entire industry. It created a space for Paul Allen to come in and step in and provide a great deal of investment funding, actually to the second place winner. And then Richard Branson to eventually buy the technology and create Virgin Galactic. And SpaceX, which you might have been hearing in the news recently. So what it really sort of speaks to here is that you're only paying for success with the competition. You're putting out there and you're saying to the world, go out and actually prove that this is possible. And you only give out the money, so the XPRIZE was only awarded once someone could actually prove that they could send two people to space. 26 teams spend over $100 million. And I think this is something that's interesting, because it's come up in several conversations that what is sort of the compensation level? And even by rebuild by design, teams spent in upwards of three to five times of what the prize money actually was. So it's a really interesting phenomenon to see that if there's something that captures the imagination, companies, individuals, organizations are really willing to invest the money, because they're becoming leaders in this industry. And what's also really interesting here is again, something that came up in our conversation with Scott and Rebuild by Design is that there's a lot of peripheral like, ongoing marketplace activity that happens with a challenge. There's a great deal of media, there's a great deal of support. And there's a lot of stuff that can happen that isn't necessarily with the challenge itself, but there's investment dollars, there's commercialization, there's new companies that come out of challenges that you may not actually have thought was possible. And so HeroX is a spin off of XPRIZE. XPRIZE Foundation runs really, really large grand challenges that run between 5 and 10 years. And they are 5 to 10 years, and they're also in the range of $10 to $25 million. They're incredibly large. They require a great deal of fundraising and support from fairly large entities. But what XPRIZE discovered is that lots of companies, communities we're coming to XPRIZE and saying I have an XPRIZE, I have a challenge I want to launch, but they weren't at the scale that XPRIZE normally launches them at. So HeroX was born to be able to provide a platform for anyone to be able to design a challenge, create a challenge, run it, accept submissions crowd from the prize, if they actually didn't have the money yet, to accept the submissions, and then to actually judge all of that in an online space. So HeroX provides a closed sort of platform I should say that allows anyone to launch a challenge. And the other thing I wanted to mention here as well is that we've heard a couple times with Jenn and Gerald about sort of the sacred nature of challenge design, and how it's an art. And one of the things that's particularly important to me, because it's part of my job and the strategic vision that I have for the company is that we actually want to democratize challenge design. We want to teach it to people. And we're working on that. We're helping our clients, we're working with individuals to design their own challenges. So we're trying to figure out what are the fundamentals of a good challenge design? What needs to be included, and then how do we help people understand what a good challenge design needs to include? And that's everything from stakeholder interviews, to judging criteria, to the rules and all the legal pieces of it. So again, just sort of a quick overview of the different types of challenges. We've got proof of concept challenges, which is like the Ansari XPrize, that you actually have to prove the impact. You have to prove that you have solved the challenge in order to win. They tend to be longer in length, as well as higher price amount, because it does require investment from the part of the competitor. Ideation challenges are smaller in prize money, and also smaller in duration. And they're mainly paper submissions. They require someone to either submit a document, a series of documents, perhaps some renderings. Things like that. But it's certainly less labor intensive than a proof of concept challenge. And so what I'm going to do now is-- on the platform itself, we encourage and we support anyone to launch any kind of challenge from any sort of industry. But for this particular presentation, I'm going to focus on a couple of cases of private sector challenges. So to really give you a sense of how competitions can be used in the private sector. And this one of our favorite challenges that's in development right now. I think it's come up in various conversations. It's called the Go fly Up Challenge. It's a $2 million proof of concept challenge to create a market for personal flight devices. So essentially, a jet pack. And an entrepreneur has come to us. She has created two companies in the past, and now this is her soul work. This is what is she is spending all of her time on. And she is developing a challenge that will eventually allow anyone to be able to strap on a jet pack and go run their errands, or go to work, or anything like that. It's a $1.5 million pot at the very end, but it's a three stage challenge, which is also rather interesting. And it's offering money at each of the three stages so that the competitors are one being judged at each of the three stages and being provided with feedback, but also getting money along the way so that they don't have to bear the burden of all the costs as they're going. But one of the things I wanted to mention with this particular challenge that I think is interesting with private sector challenges is that Gwen will be seeking the IP or a royalty-free license to the IP from the competitors. Not just the winner, but all of the competitors. Because what's interesting is that if you recognize if you only take the IP of the winner, it's actually motivation for everyone else not to win, because then they get to keep their IP and they get to commercialize on their own. So instead, the agreement has to be framed to take the IP of every single competitor who enters the competition. Which is why she also has the $1.5 million grand prize. When you take the IP from someone, you've got to compensate them really, really well. And so instead of taking full IP, she's going to do royalty-free. So she has a period of time where she can use the IP license so that she can keep the prize amount just a little bit lower than what it would need to be. And then she's also got investors who have created the pot of $1.5 million. She's not putting all that money in herself. So it's an interesting set up as well, because those investors are getting-- will get part equity into the company that she will eventually create with the solution that comes out of this challenge. So some really interesting, groundbreaking things that will happen here. So the MNP Challenge is run by a consulting company out in Calgary. And they're running in ideation challenge to improve health outcomes for Aboriginal Canadians. And what I really love about this particular challenge is they've really gotten stakeholder interest, not only from First Nations communities all across Canada, but as well in their own company itself. They've really gotten the right stakeholders involved. And also, I took them through about 10 weeks of actually designing their own challenge, from teaching him how to do stakeholder interviews, and talk to subject matter experts. What kind of questions you need to be asking, how do you set judging criteria, and what do those criteria need to be. What are the rules of the challenge? And another really great piece about this challenge is that they are awarding a $25,000 grand prize, but they're also going to open the challenge to a People's Choice Award. So they're going to do a public vote. And they're going to ask the public to vote on a winner and award them $5,000. And if you can also see on the dashboard here, of the challenge itself, you'll see that there's entries. The challenge opened probably about a month ago. So it's really quite early to be getting entries, but this challenge is also structured so that all entries are public. Which means that throughout the process as people are submitting, you will see all of the entries. People can comment and then they can go back and actually update the entries and make them better and stronger. Rehabs.com is another really great challenge. This is a company that is focused on supporting individuals with addictions. And they are running a video challenge, which when you think about submissions, you think about paper. And so the actual product in this case they're asking for a video that people will submit, that in their hope will destigmatize substance abuse and behavioral addiction. And they'll award $10,000. And what's particularly unique about this challenge is when you think about-- they want people to actually prove that they've had impact with these videos. But how do you do that? How do you get an unbiased judgment of someone actually saying a video is great? Because all you'll do is ask all your friends and family. And they'll give you great comments and you'll only ask the people who are going to tell you that the video is fabulous. So instead, they are going to run the proof of concept, which means that people will submit the videos, and then they're going to go to a closed group of 10,000 people of their own community, design a survey, and ask that community of people who have an unbiased view who don't know the competitors to then demonstrate what is the impact that's being had by these videos and how are you reacting to them? So that should be launching in the next-- actually in June. The Ellucian Student Success Challenge was a $50,000 ideation challenge. It was actually recently awarded. And what's particularly unique about this challenge is Ellucian came to us and they said we want to run a closed competition for just our customers. We don't want to include the crowd, we don't want it to be open to everyone. We want to run it just for our customers because we're really interested in the issue area of student success, and we want to understand how all of our customers, who are all higher education institutions, are handling that issue. So again, we fully trained their staff on how to design their own challenge and then how to run it. And then they completed it on their own. And the winners, as you can see, were recently announced. And then finally the Goldcorp Challenge. Another unique challenge. It's just sort of in the realm of challenges. It's not a HeroX or XPRIZE challenge, but I'll just point out here that-- so Rob McEwen had a gold mine. He couldn't find the gold anymore. He's like, what do I do? Like, I'm going to lose all this money. I can't run my business anymore. What should happen? So he put out a challenge for $575,000, and was like, guys go out and find me some gold. Like, figure out what's going on my mind. Why isn't this working? So 1,200 teams from 50 countries-- if you can imagine, all the goal devices or something like that. They did more than that, but they figured out how to find six million ounces of gold, valued at $6 billion in his mind. So that's pretty incredible. So Rob only had to give out $575,000. But had found $6 billion of gold in his mind. So can you imagine what will happen-- what could happen with the potential of challenges? And then finally just a couple more that are happening in the world. Some of you may have heard about the next Netflix prize. This $1 million. And what's really interesting about these challenges is that they're all very, very specific to their work. To their industries. Netflix they didn't want a philanthropic challenge. They didn't say go save the world. They said we have an algorithmic problem. We want to improve our business. Come in, we'll give you a million dollars, and tell us how to fix it. And the same thing with Allstate. They said we have an algorithmic problem. We want you to solve. We want to increase-- we want to improve our customer service. And we'll give you $10,000 for it. So look at the number of teams that participated. 20,000 teams in Netflix. And 1,568 teams in Allstate for just a $10,000 prize. So people are really interested in challenges. And you give them the opportunity, you create compelling guidelines. And you give them something that's really interesting and visionary to work on, they will show up and participate. And I mean these are all just sort of demonstrating that it doesn't have to be around things that further education and health around the world. Those are all really, really great, but these will also do that, because these are software things, these are coding things, these are all sort of solutions that will then be applicable hopefully to other things. And just a few last things that I'll mention here is I'd mentioned with the Gwen-- the Go Fly Up Challenge that IP is something that's really interesting that we haven't talked about yet in the past day. There are ways that you can structure IP with the challenge. XPRIZE allows all their competitors to keep their IP. But you can consider other options with challenges. That perhaps you want to split the IP with teams, or you as a sponsor want to retain the IP. So there's lots of interesting ways as a private sector company, that you can look at what a challenge can produce for you. And then finally I just I wanted to point out we run challenges. I spent my time here talking about private sector challenges, but the platform is open to any kind of challenge. But I did want to demonstrate to you that all of these municipal challenges are all also partnerships with private sector companies. So for example, the Innovate San Diego Challenge is a partnership between the UT, which is the largest newspaper in San Diego. Private newspaper in San Diego, and the mayor's office. And they've come together to say we want to do something really amazing for San Diego. How do we make that happen? With San Antonio, the same thing. The gentleman who offered the prize is the CEO of Rackspace, which is one of the largest cloud computing companies in the world. And he came together with the city of San Antonio to sponsor that challenge. So just a demonstration of lots of different ways that the private sector can be engaged, and different kinds of challenges. And we hope you'll come and check out some of our challenges, and perhaps launch your own. Thank you. The good news about being last is-- oh, I have to stand up here? [inaudible] Is this working? The good news about going last is I get to hear what everybody else said. The bad news is that my slides are not that relevant anymore. But I think the first thing I'd say is that I come from-- it is a possible turn the lights up? Because I'd rather have more of a conversation. Is that OK? Have more of a conversation. I come from a different place than what you've just heard. You have government, you kind of have private sector focus. I come from a vocational technical school down the street called MIT. And so when we look at this thing much differently, as you're going to see. And I'll tell you about my experience. What is a goal that you all have about when you want to run competitions? What's the goal that you want to have? Better ideas. OK, what else? Anybody else? You've just heard-- yes. No, better ideas, new participants to find, identify, and cultivate new talent. Yes, what else? Public education. Interesting. That's where I'm going to go. What you've heard here is they want specific outputs. They want specific outputs. Those were what those challenges are. What we do is we have a longer term view. We want to build an ecosystem. We run competitions to build ecosystems. And I would argue that neither is better than the other, but if you have the time, if you build an ecosystem, you will get lots-- that will pay dividends over and over again. So what we look to do is not to run a competition to catch a fish, we run competitions to figure out how to teach people how to fish. And I think that's a subtle difference as I go through and I talk about it. I think that's really, really important. And so if I go to this, which we have a remote, do we? We do? This is forward. We have remote one? So innovation is this thing that we're trying to promote, rather than a specific outcome. Innovation. And it's this new cool thing, but do we really know what innovation is? It's interesting, innovation-- everything is innovation. The Wall Street Journal did this, or somebody did it. It was reported in the Wall Street Journal that innovation, the number of times innovation is mentioned is just spiking. It's exponentially growing. And is a new Ford F-150 that's made of aluminum-- is that innovation? We could have a long debate about that, but the term is not particularly well defined, which bothers engineers like myself, even if I'm a fake one. I'm a fake one because I got my engineering degree at Harvard, which they think is hysterically funny at MIT. But these are the things that are not true. Competitions-- you've heard this from the first two-- competitions are not a panacea by themselves. They are not a panacea by themselves. The idea that eureka moments happen when you study innovation, that's not the case. There's actually-- you can trace these things and see them building up over time. And there is kind of a moment that seems like a eureka moment, but that's just not true. And then the other one that kind of come back when it swings back and says, oh competitions don't work when people think they're a panacea is they're completely useless. And none of these are true. None of these are true. What is true is innovation is a hard, long process that is fostered through a complicated system. And that's not what we'd like to hear. We like simple answers, but that's not the way it works. The reason why Silicon Valley is Silicon Valley, there's not a simple answer to that question. The reason why MIT is MIT, there's not a simple answer to that question. It's a complicated system that needs to be built, and there's not one single element to it that makes it. Competitions are one tool in that that can facilitate that. And if you put a competition into a system that's ready for it, it is a very valuable thing. You could take that same competition and you could put it into another system, and it can not only be not effective, it can be counterproductive. So it is not a question of a competition is useful or not, it's a question of where are you putting that, what context, how ready is it? And then there's actually a growing body of work, some of which you've heard here that's studying of the role of competitions. So what I would say is that-- this is why I asked you. It's like the first rule of engineering is to find your terms. And then the second rule is define what you're trying to solve here. What are we trying to solve for? And a lot of times people implement competitions, and I say, what are you trying to do? Are you trying to create new companies? Is that the role? That's a vanity metric. That's a metric that people love to say. How many companies come out of the MIT $100K Competition? I think that's the wrong question to ask. I think that's a fundamentally wrong question to ask in our case. It might be if XPRIZE sponsors one, that is a very valid question, but for us, that's not a valid question. The question is does it do something to accelerate the ecosystem and the production of entrepreneurs that will be successful? Do we accelerate the process to teach people how to fish? The answer to that is in the right circumstances. But even if you do that, when you're trying to teach this, you can teach people innovation, but it's about, as we say, the head, the hand, and the heart. It's the head you have to have knowledge about how to do this stuff. If knowledge is not sufficient, you have to then put that into practice with hands. And then you also have the right mindset. Often you see people that changing the mindset-- a competition can go through and produce no companies at all-- but if it fundamentally changes a mindset, this will pay dividends over and over again. It knocks companies, organizations out of their homeostasis. If you can start to create a different feel. It can identify talent, as someone mentioned here. This actually is very important. I work with companies and they can run a competition, there are going to be no companies that come out of it. I mean, you just have to understand that it's very unlikely that companies will come out of it. And often they vastly overestimate the value of the ideas that they have. It's like, if you have kids, every parent on the sideline thinks that their kid is just amazing athlete. And likewise with companies, the companies think that their ideas are amazing. And you see this over and over again. And then they set up venture capital funds to promote their own ideas, and they don't survive in the marketplace. But what you can do is identify talent, and then you can also use competitions to build bridges. You can use competitions to improve the ideation process, which you've heard about from some of the previous speakers. And this is partly changing mindset, but it's also training them in the ideation process. And it can be a training vehicle to identify them, and then use them as a carrot to get them to do certain things. I want to win the Boston Marathon. I'm not going to win the Boston Marathon, but if I try to win the Boston Marathon, I might get in good shape, I might be healthier. There might be a whole lot of ancillary benefits. The competition actually is a head fake to do other things. And fundamentally, going back to this, changing the mindset-- changing the culture is I am embracing the fact that you can be entrepreneurial, you can fail, you can do some of these other things. That can be important. But it doesn't do it by itself, as I'll talking about later. The old tools, when I worked at IBM, is anytime we'd have a program, we'd have committees that get together, and the come up with the program du jour. Here is our invention program. Suggest an idea. Our suggestion program. They'd change the incentives, but they didn't fundamentally change the culture of the company, and they didn't identify talent, and it was a particularly systematic. And it would go away. Today, competitions, startup weekends, hackathons, incubators, mash ups, ask me anything types of things, online things. These are just some of the tools that people use in the entrepreneurial space that have actually turned out to be very effective to knock people out of their homeostasis and start to achieve some of those things that we talked about before. Now does this effective in large companies? And the answer is, it can be. It can be. And the XPRIZE has done a lot of work on this. Georgina Campbell, a TPP graduate, and professor Fiona Murray wrote a paper, which I can make available to you as well. I wrote a paper with what some large companies have done using it. I was very involved in the Clean Energy Prize, and I'm very involved in $100K. And I want to just talk about that. So the Clean Energy Prize we set up-- and it's been phenomenally successful by almost any metric you want to measure by. And the one that I don't want to measure by is a vanity metrics. And that's what it always gets measured by. But it's produced enormous number of companies. But what's more important is that it produces these entrepreneurs that go out and then start the companies. And their first company often fails, but then they've learned the process, and they're onto the next one. But the Clean Energy Prize was one part of an overall system that we put together that included a course up there-- I'll go back to energy ventures. The Energy Initiative came out with Ernie Moniz at the time, there was a student driven club, there was a social place called the Muddy Charles, which was the pub where people would go and they would have serendipitous collisions. You had our Center pushing it. You had Greentown Labs which was getting started. You had the energy conference. And within this, the competition was one other catalyst. And it connected-- and we designed it such that after you took-- you could get all this stuff and you could build your team and you could take the early classes. But if you were good, in the second year you were there, you got to take this energy ventures class. And the energy ventures class was in the fall. And then if you did well in that, then you get to rebuild your team and then you get to go into the Clean Energy Prize. And this was kind of the rhythm that we went through. The interesting thing is on the prize is that there was something mentioned here. The people who win the prize actually are not the ones who usually do well. And I'll give you the story of John Bloom. He said one of the critical moments-- he has a company called Podimetrics-- he said one of the critical moments in the history of our company was when we did not win the $100K Competition. Which is really interesting. And I want to write this article and this podcast about-- you should try to enter a business plan competitions, try your hardest to win, and hope that you don't win. Because what happens was is that when they win the competition, they don't deal with the most fundamental issue that's-- to be successful. And that's the people issue. That is, do you have a strong team? Because things are going to go wrong. And in the case of Podimetrics, when it didn't succeed, they didn't win the competition. All the people who weren't committed to it left the team. And only the core people were left were able to deal with the hard part of failure experimentation, getting knocked down, not being able to make it, and having people ridicule them at times, were able to force through the company to be very successful today. And that creates an anti fragile organization to steal-- to lead term which I like a lot. Is it you need to create anti fragile organizations to be successful innovation? Those are ones that can fail and get stronger from failure. Not just absorb failure. They get stronger from it. So competitions, it turns out that not winning a competition might be the best thing possible. And historically, Akamai, Hubspot, Guitar Hero, these companies came out. They didn't win the competition. And then if I show you a list of people who won it, they generally aren't the ones who do so well. So the advice is a prize is a head fake to-- and you have to understand your goals with what you're trying to do here. The person who wins the prize in our scenario, which is different than the scenario you've heard here. It doesn't really matter. What matters is, is that these people got in shape, and they got healthy, and they started building teams and relationships. And this is going to pay back over and over again. Thomas Friedman said, what's his most his favorite energy source? And he said, my favorite energy source is the human spirit of innovation, which is 10,000 people trying 10,000 different things in 10,000 different garages. And that's what makes for an ecosystem. It's not the singular idea. It's this continual process of innovation and anti fragile human beings that are connected to each other. So you have to, if you run a competition, build a supporting structure before, during, and after the competition. Otherwise you're basically wasting your money. The first year we ran the Clean Energy Prize, it was not worth it, even though we had all the support structure. The benefits came in year 2 and year 3, when they started to anticipate it, they started to take all these courses, and move their way through. And I think that's the lesson that we've learned and it's really paid off for us. There you were. I thought you'd be over there. So let me just summarize is that as I've gotten older, I play basketball. And a colleague of mine, we were sitting there after one day, and we were like, man this is so hard to stay in shape. And it takes so long. And he said look, anything that's good, takes a long time. Bad stuff happens really quickly. Like, if something happened quick, it means you blowed out your ankle or your knee or something, and you're not playing anymore. If you want to get shape, it takes a long time. And this reminded me, there was this famous person, this is Red Auerbach, and he was the most successful basketball coach ever. And he's saying to his team a year is not a long time. So when you're thinking about these things, you cannot just think in the short term. You have to think about, are you building the ecosystem? Are you building the capability? Are you teaching them how to fish? Is that what this new tool to your tool chest is doing, that you're pulling out of your tool chest? And if you do that, then these things are incredibly valuable. If you think that they're going to just be the quick fix, it doesn't work that way. So that's a different-- by the way, I just go back to what I said at the beginning. I have a different perspective here. And I have the ability to do that because I come from a place that has a very large endowment. Very small compared to a school at the other end of Cambridge, but a pretty sizable ability to think long term. So I have ability to say these things when sometimes other people don't. [applause] This is quite amazing, especially after this morning. So we go from learning how to fish, to learning how to fly, and then we're going up into space. I thought it was rather entertaining session. Thank you very much. Jenn referred to it very briefly, early on in your presentation. And I think it's important for us to learn more about how do we measure and how do we judge? And as you realize from our conversations, and learning a little bit about how the design fields operates, our jury processes are often very subjective. I've been on jury-- it was actually not for a Van Alen Competition, in which one of the deciding factors was a photo of one of the main competitors who had such a fancy suit, that he didn't make it through the end of the process because people didn't trust the suit. And that's of course a very sad thing if that becomes the reason or making decisions. So help us think through how could we make the design fields-- because in the end I hope we can get some great advice from all of you, because you're very creative and very smart, how could we get to design fields to become a little bit more objective in our decision processes during a design competitions? So we had an interesting discussion at dinner last night. It's an analogy you might want to tell, but I'm going to steal it. And then you can tell me whether or not I said it right. About figure skating competitions. So, if you think back 20 years, at the end, the judges would issue one score. And the Russian judges would always give the Americans a horrible score, and the Americans would always give the Russians a horrible score. And there's this nationalism thing going on at the same time. But people got really upset because they felt like they were being overly subjective, and obviously biased in the way that they were judging these somewhat subjective-- I mean it's performance art-- so these somewhat subjective competitions. And so more recently they changed judging slightly now, where the judge has to-- on that triple axle-- at the moment the triple axle happens-- vote on how good that triple axle was at that moment. So the judges are voting like 20, 40, 50 times, over the course of a performance at the moment. So they're not seeing the whole thing in the context at the end. It's more their forced at that moment, to judge the quality of that particular action. So in some ways, that's a way that in figure skating, they've tried to force more objectivity in moments of the overall design evaluation process, that I thought was just fascinating last night when we chatted about it. Great enough example that that is my answer for you today. Because I thought it was very good. It is a great one. And another movement example. So good. Any more suggestions? Yeah, so one of the ways that-- so obviously this is something we have to teach people. We're trying to do it ourselves, but we also have to support clients, organizations, companies, individuals to really understand what it means to put judging criteria on a competition. And so what does that mean exactly? How do you do that? Like, what do they have to look like? And sort of the simplest way-- we always like to do things the easy way at HeroX, and so the simplest way that we really describe it is that you come up the criteria, and you put a number against it. Unless you have a number against it, with a criteria, it doesn't count. Like, eliminate it. Like, that's it. There's actually room for subjectivity in every single submission, as there should be because a judge is bringing a certain level of expertise to that submission. But your score card, which is what we call everything-- a judge has to have a scorecard, it has to have a very specific criteria, and it has to have metrics that are put against it. And unless it entails someone actually meets those metrics, they're getting a big fat zero. And so that's how we're supporting individuals to really sort of-- our companies to understand what's required in just a really, really sort of simple, simple way of what judging criteria needs to look like. I'm going to react with one thing to that too is that there's this horrible Goldilocks part of this whole process too. Because you can, if you overly specify the requirements that you're trying to get people to meet, if you set the baseline, if you set the number, they will design their solution that baseline of that number. So you will have people that gain their solution just to make sure that they're checking all of the boxes exactly how you say that-- and you could end up-- especially if you don't fully do problem decomposition well at the beginning-- you could end up getting exactly what you ask for, which is not what you want. Yeah, which is why the framing the problem is so important. Because you have to understand-- which is why the whole problem definition is so critical. Even so when we talk to customers is that you can't design a challenge in a vacuum. You can't just decide that this is what the world says, and this is what I want it to be. You really have to go talk to the world. You really have to go talk to the stakeholders and the subject matter experts who will tell you this is where the bar is, and this is where we think, for a million dollars, you can raise the bar to. And what we generally sort of encourage people to do, and even when we're designing challenges, is that OK, the bar is here. Raise it, five times. What's really critical with competitions is finding that balance between what's feasible and what's the grand vision. Because that's where you're going to sort of not get people to just meet a minimum threshold, but really sort of strive towards something that's much more visionary, and something that's going to revolutionize the space. So I wanted to amplify what I thought was actually an interesting different point of view. I think Bill was very nice in the way he presented basically, his fundamental disagreement with what the two of you do. Now it's find me night. Now it's fine to disagree, it seems to be with Nidhi. She's the private sector. If she wants to do that and make money with HeroX, getting people to use their platform, that's fine. You can't stop that, or whatever. But when it comes to Jenn, who's using our tax dollars to pay for all this sort of stuff, we have more traction with saying, is the federal government doing things in the right way? Now you've said prizes are a head fake, as you put it. But Jenn and her colleagues are not head faking. They're not using it that way. They are using it for the object, for the output. Would you be saying to Jenn-- and you can look that way if you wanted to-- that the national government should instead, be doing challenges and prizes to develop sort of entrepreneurship or innovative types of people and companies-- not one object, but this sort of helping them-- giving them encouragement to learn how to fish, or learn how to innovate. And indeed, the national government wants innovative people, because then they're innovative companies, and that produces wealth, which then can be redistributed in very positive ways. That's the system that we have. So isn't your advice really to Jenn and her colleagues in the Congress and at the White House to sort of recast this in her remaining years there, before she goes back to NASA, or into the private sector and joins HeroX or wherever she finally goes? Just to clarify, everyone knows he was trained as a lawyer. And he's led the witness. But no one objected. Nor did I object. You're absolutely right. There's a fundamental disagreement. And I don't know if Jenn can to do this. This is a political process where you have things like Solyndra singe, which was just in my estimation, was absurd. You're in a political process where you get punished for failure. Let me tell you, the easiest way to-- first of all, the easiest way to have no innovation is to do nothing. I'm sorry. The easiest way to have no failure is to do nothing. If you do nothing, then you don't fail. And that's what happens in many government positions. Not Jenn's, of course. But government people are frozen because they don't want to be seen as Solyndra. You cannot innovate without failure. So we're in a political process where this kind of stuff happens. And people are under the microscope. Absolutely. I mean if you think about this, we should be creating entrepreneurs and less focused on that one thing. And if you persecute people for one failure, you're just going to kill the system. You're going to kill innovation. And so frankly, I'd give up on government in these types of things. They have very, kindly taken the Clean Energy Prize, and they've rolled it out across country. But if you come into my office, there's a sign. And it says, people before projects. When we analyze something, everybody likes how big is the market. What's-- let me show you. Because they can measure that. It's much harder to assess, has someone growing? What's their trajectory that they're on? What capability do they have? How do you assess someone's entrepreneurial spirit? It's very hard, but there's a group of people who are doing this. How do you do it? Well, there's a whole group. If you go to eef.io, we just had a webinar on this. But could you give us, at least, several sentences? 140 characters? Yeah. So first you measure the head, the hands, and the heart, basically. You could measure knowledge. And that's what people usually measure. How much do you know? But then you have to measure their capability. And then you have to kind of measure their mindset. What's their mindset? And so there are methodologies to do that. But they're developing, and they're very hard for people. And then why you hire them. Is that the idea finally? Or you give them a prize? What are you doing with all this? I mean, if your output is-- We're in the business of educating. We're in the business of creating entrepreneurs. And if you create entrepreneurs, then you will get companies. Full stop, that's what you want. That's why when I said don't worry about the specific company. If you create an ecosystem, you will get innovation after innovation, after innovation. That's my point. But I'm just wondering if you're marrying the idea of competitions to create and encourage innovators rather than a product. Or you're saying no, I don't even want to use competitions for that. We use competitions to try to produce role models, because culture essentially, is the operationalizing of values that you have. And to do that, it goes through what urban legends you have, who are your role models, what do they see day in, day out. And let me add that I object. To the lawyer? Yes. If I could go back to the one slide that I had on outcomes. That particular list of things that was in the Deloitte report, lists six different ways that the government is using prizes and the public sector is using prizes. And a couple of those outcomes are very much around kind of changing behavior, market stimulation, and market simulation it gets down at the entrepreneurship level, to the business creation level, to the supply demand advance market commitment kind of level. But learning from things like what MIT has done with MIT Clean Energy Prize, the Department of Energy scaled something called the National Clean Energy Business Competition that's now in six regions across the country. That's all around trying to encourage clean energy companies developing kind of across the country. But I will say that that one tends to be evaluated at a different metric level than yours. You're completely right in that because the federal government is investing in six regional business plan competitions. The metrics that they're driven to be evaluated by are things like these vanity metrics, like companies. As opposed to when an organization like MIT, that can have more of a kind of system hundred year look, 50 year, 20 year outlook things, can say truthfully, the right metrics that we should be looking at are this set, not the vanity set. And so I think that there's a similarity in philosophy. But sometimes the way in which you have to justify and continue to keep support for the programs that you're doing, you have to nail on some of those metrics that mean something to the appropriators in order to keep support going for those. Even if the evil plan under the surface all along is knowing that things like what he has mentioned are really more important. And that's almost like the wolf in sheep's clothing bit of entrepreneurial competitions within the public sector and the federal government. I would say we still have a lot to learn from folks in the private sector that are able to do those in a very, very impactful way. By the way, your objection, I completely agree with. And I think what happens is, we can agree at this level, but then when it goes up to the funding level-- this is the whole Solyndra thing. The Solyndra discussion about Solyndra was absurd. I mean, it's like so many discussions at the government level now. It's absurd. Can you describe a little bit, for the audience, more for those who don't know about the Solyndra. So Solyndra, they invest in clean energy, and Solyndra fails. And everyone says look at this. They spent money on it. It didn't work. Someone has to pay for this. Well that's what happens in start ups. There is no such thing as innovation-driven entrepreneurship without failure. It does not exist. It's like a chocolate lava cake that tastes good with no calories. It's a complete figment of people's imagination. That doesn't exist. So they were waiting on the sidelines for the Solyndra-- whether it was Solyndra or something else, they jump on it and say see, this program doesn't work. Well guess what? My first company failed too. You could have jumped on me and said no, nothing ever happened. But that's how you learn. And you go through the process. There's an old saying, which isn't technically true, that lemons ripen before plums. Bad things happen before good things. That's an experiment. And if you prevent people from experimenting, you will kill innovation. And that's the whole-- that the Solyndra story. I'm still trying to grapple with this notion of using a competition mechanism, not to procure a jet pack, not to procure a design for a building, but to procure innovative designers, or to procure innovative entrepreneurs. And that's what I want to hear a little more about. How you structure that as a competition? Or are you just talking about not a competition, just investing in and helping with encouraging that kind of development, but not using a competition mechanism to actually procure that object, which is not an object of a jet pack. I think we're in agreement here, fundamentally. There's different kinds of competitions. We've heard that all day long. There's just different kinds of competitions, and what they produce outcomes for. So what I'm asking all of you now is, starting with what Bill sort of brought to the table, but how do you use the competition mechanism to actually choose or select innovative operators, whether they're companies or individuals? So I'll just give you an example. I worked for Ashoka for a number of years. And for those of you who don't know, it's one of the foremost organizations in the world, I would say, to support social entrepreneurs. And the way that we ran competitions was as Bill is talking about, to support entrepreneurs. Because they were solving problem-- what they were doing is sort of telling us what their companies, what their organizations, and what they as people were doing to solve those problems that were being proposed. They weren't going out there and inventing something new, or creating a new product, or anything like that. They were telling us as social entrepreneurs, how are they solving this problem. And the goal of every single one of our competitions at Ashoka, was to then give those prizes so that people could grow those companies. It wasn't for them to go off and do something else new. It was to grow something that they were already doing. So the focus of change makers competitions was fundamentally to support entrepreneurs and people was a huge part of that. Because part of the profile that you submitted was who's on your leadership? Who is advising you to be a part of-- to grow your company? And so it was a fundamental component of it. So you had an open kind of thing where people would apply and say, I want to be a change maker and win the competition that Ashoka is running. And then Ashoka would have experts at Ashoka to judge all of these people, and then finally choose. I'll give you two examples. One would be the Knight News Challenge that targeted innovators in the media space. And the second will be the Georgetown Energy Prize that actually targets communities and not individuals. But the Knight News Challenge, it is an ideas competition where they seek the best ideas for integrating data-driven journalism with the future of media. But they use that challenge in order to identify the people with the best ideas. And then much of their programming after the challenge is actually to support that class of people that won the Knight News Challenge as folks that could be potential future media innovators. And so they use the challenge to identify the folks with the ideas, recognizing that the idea itself, might not be the thing, but that cultivating the person who had the idea will create potentially longer term dividends. And to pivot just slightly on the Georgetown Energy Prize, I find that one to be fascinating. That's another university-sponsored prize, just like the MIT Clean Energy Prize where they have launched a prize where it's around trying to get small communities. Communities between 5,000 people and 100,000 people. So not like Manhattan or Boston, but smaller communities around the country. And listed to compete against each other for energy utilization at the local level. And so they're trying across the country to reduce energy utilization and increase energy consumption efficiency by enabling communities to take whatever approaches they may identify at the local level to create behavior change in communities ultimately for greater good across the country that then lead to best practices that other communities could adopt across a diverse set of small communities. So in that case-- and I bring this one up because we talked about product and outcome, and we've talked about people as entrepreneurs, but there's this whole other set of challenges where you're actually looking at using them to try to cultivate behavior change amongst mass groups of people. Really, that's the basis of things like the Spelling Bee. I mean, Spelling Bee has been around forever. And that gets a lot more kids spelling for the purpose of competition. Same thing that-- I think it was back in the '80s, when Reagan had the Get Fit thing and Clinton had it where everyone had to run a mile. If you ran a mile once a week when you were in school, you got the president certificate that said like, you're recognized for the President's Get Fit. That was just to get them fit. But a whole class of kids to get them fit for some small, small task to change the mean performance of a large group of people. So you can also use challenges for that purpose. Can I just build off that? What we did-- and I set up the Clean Energy Prize as a forcing function to make people use all the other resources that we had in place. And the question was, how do we get them-- and how do we get them to go-- how do we do that? Well, they realized that if there was this carrot out there, and it was $200,000-- oh, $200,000-- so it got them to go form teams. It got them to take the classes. It got them to do the things that they needed to do. Just like you're saying is, they got in shape, basically. And all these other elements that were there got utilized. And so was a forcing function to get them to do that. And then at the end, when we give the prize out, I say, I'm sorry this was a bit of a fraud. For those of you who win $200,000, you're going to find out $200,000 doesn't really-- doesn't mean that much. Yet another head fake from MIT. But at that point, they get it. Because now they realize that they've learned so much. And it's not just learned so much is they form teams. I mean, there's this whole thing that innovation is this individual, eureka moment that happens with one person. That's just-- the technical term is bullshit. Innovation is a team sport that happens, and it's kind of a process that you go through. So them getting to form teams between the technical, and the design, and the business community, this is invaluable for them. And that forcing function of the competition is really-- I think what you're hearing is that's the value, and that creates the capability which is more important than the fish. I was going to ask like how do you-- Who's the moderator? Is he the moderator? [interposing voices] It's a challenge. It's a challenge, not a competition. We're looking at each other, trying to determine who's going to speak. We have this whole winking system going. If we talk about the collaboration, how can you take your collaboration systems to the design field? Because in the design field, so often it's still the designer that wants to lead the teams, for instance. But I think when we look at your projects, it's actually very interesting that sometimes they're led by the oddest person in the room, the wedding dress designer perhaps. And I wonder if that's something that we want to stimulate more in our field. And this is a key insight that I took from your survey too. When I looked at when you were presenting the results, and I was skimming it seeing how many folks collaborate with folks outside of discipline in design competitions, it's alarmingly low how much collaboration there is. And I get the lack of collaboration, and I get that architecture is a specific field, and landscape architecture is a specific field, but there is very little desire to it seems, and also in practice, collaboration cross field. Which is one of the things that I think-- I know within our prize space, interdisciplinary collaboration and teams are key to getting the breakthrough ideas and approaches that might be a traditional approach in this field that just is not something that's in your gravity field, in the way that you are traditionally working. And one of the things I think Rebuild by Design did really well. Forcing folks to consider bringing all these different skills to the table that might have seemed like cognitive dissonance, and we don't speak the same language, and this is not the way we like to work. So there could have been. I'd be really interested in talking to more of the teams that were involved in that, to understand if this truly was the case. That's uncomfortable. It's uncomfortable to work with people that have cognitive diversity from you, and don't look at problems the same way. But that's also a lot of the value that you get from going through that tumultuous, forming, norming, storming-- or forming, storming, norming process in order to really allow more innovation to the table. Let's just clarify, because the point-- I think it is 46 point something percent. So that is actually within the design field. So designers do tend to work with other designers. But then one thing I didn't mention this morning, and this was quite interesting, the one field that designers would like to work with-- and I was like oh, really? There's so many fields that you can work with, and this will be a nice collaboration of course, and interesting, but it was with artists. But there's so many other fields that you always want to collaborate with I think. Yeah. So what's fascinating for us sitting here listening to this, if we have a particular takeaway from this discussion, has been this notion of using the competition mechanism to procure or reward changed behavior, or behavior that we think would be better i.e. for the design world, somehow being able to have a competition in which we end up choosing and rewarding designers who are doing their practice in a better way. It doesn't necessarily have to be in a more innovative way. That word, as you've said, is overused. So but in a better way in some way or another. And somebody has to design that brief, develop the metrics of determining what that is. Stating what the goals actually are for that as a designer, rather than the product itself. And that's an interesting challenge and an interesting thought for all of us. [interposing voices] Can I just say one thing? And this won't make the room happy. But we're not a design school at MIT. And Boston isn't a great design-- That makes us happy. Wait. Hear the whole thing out. But we have traditionally been very-- They're both designers on the faculty. So traditionally, MIT has been very good at entrepreneurship because the team basically is made up of hackers and hustlers. You've got hackers who are technologists, hustlers who are going to get the business side of it together. And they do really great stuff. Now the problem is, they say oh, look at New York. They're taking off. Well what happens in New York is a stack of a company, you don't have the hackers and the hustlers as much anymore. The hustlers are square, the hackers, the techno people are much less. But you have this new breed that we don't really understand called hipsters, which really are kind of this user experience thing. It's like whoa, how do you do that? The other design-- And that, in essence is-- when people say, well New York's in competition with Boston. First of all, entrepreneurs love other entrepreneurs. Anyone who is an entrepreneur is our friend. The enemy is clear. It's Goldman Sachs. But we don't have the same entrepreneurship here as New York. New York is much more e-commerce, fashion, kind of user-experience oriented. And we're trying really hard. So we're trying really hard to incorporate the hipsters at MIT. Let me tell you, it's hard. It's hard. Because hipsters are divas. They're harder to work with. Engineers are kind of weird, but eventually you get them to work with the hustlers, and everything kind of works out. Now you have a hacker hipster hybrids too. [interposing voices] But this idea-- and I'm dead serious about this. We have a problem incorporating designers. And they say, the only thing that matters is design. And the engineers said, wait, you just convinced me that engineering wasn't the only thing that mattered. And we have this triple helix going on. I'm telling you, the hardest thing is to get the designers to be a part of the team and recognize a proper role. They think that they should run the whole thing. So if we agree with you finally, we will design a competition that will reward those designers who now take the leap and go beyond the old engineer and become the new engineer-- the new designer as you're describing. And that would be part of this kind of effort to signal to the marketplace of designers that they have to be different. We've gotten our time signal, which means that we turn to the audience now. So questions from the floor. Right here is a question in the middle. He'll take both microphones. When are you going to come out with a competition for competitions, because that's really what's missing here. We did that about six or eight months ago where we did exactly that. We were like, we created a challenge design for you to design a challenge. And then we had certain criteria that was like, what makes a good challenge? And we created all the criteria for what are the fundamentals of a challenge? What makes a good challenge? What are the components that it needs to have? And then we ran that challenge about six months ago. And I'll speak at it from a slightly different angle where-- and this is more my NASA hat on than the White House hat. I was charged with trying to get at NASA more people to use prizes and competitions in their portfolio of tools to solve problems. So contracts sometimes are the best way to do business, grants sometimes are the best way to do business, partnerships sometimes, cooperative research and development sometimes, and prizes, in some cases, for the right types of problems, should be incorporated in your portfolio. It was like hand to hand combat every time with program manager, trying to get them to understand how this particular type of method that was new to government and didn't have a 20 page manual of the way that you did it. Which also squeezes the innovation out of this particular tool that is so flexible now in government. It's like, guys you don't have to follow a very specific process for this. That's what's desirable about this tool is that you can customize it to your needs. The most difficult thing for me was trying to get folks to even understand what types of problems where well suited for challenges. Let alone a challenge for challenges. I mean, if I were to just ask-- and this is an organization that's done 10 years of challenges, and over hundreds of them at NASA, and I couldn't still do a wide call to say, hey guys give me your best ideas for challenges. Because most people still had no idea what a challenge was. They had no idea what kind of problem was well suited for that type of mechanism. And then they would think that a challenge could either do way more than it actually could do, or they'd just do it for video competition, or for an apps competition. And not actually be thinking about the things in the middle that are the highest impact. So so much of the work that we did was literally hand to hand-- like hand holding with program managers to help them figure out what types of problems are better suited to that, and then helping them to pilot them. So I struggle with this whole notion of challenge for challenges, because I don't see high quality coming out of challenges for challenges. Because folks just don't really know. And if they can do this, I think it's going to be awesome. I do. I want them to succeed in being able to equip folks with the skills to be able to generate their own challenges, but it's hard. It's hard to develop a good challenge. If I understood Bill correctly, you had a system where you run from the challenge to challenge, and I could kind of like that. So it gives you a way to rethink what you're doing in the middle of the model. Would that have worked for Jenn? From challenge to challenge. You mean that we go-- The full circle and-- Yeah. So we have a class, by the way, that goes with this, sponsored by the XPRIZE. How do you build a competition and how do you do it. But I just think it's really complicated problem and has contextual energy that's different than the app world, and things like that. And I think it's hard to have a general solution here. You can have a certain mindset. Now I'm from New York, so I'm born cynical. If we go from challenge to challenge, you just have to have a long term approach to it. But I think just one, really really quickly thing to add to what Jenn was saying is that we're a start up. We've been around for about a year and a half now. We're going to fail. Like, this is just inevitable. Like, we're going to have moments of failure where this isn't going to work. But I fundamentally believe, I come from a design-thinking methodology idea world, frog design kind of situation. And so I really do believe that these are methodologies that we can teach other people. The high quality is not going happen right away. It's just not. Like, we've seen a lot of shitty challenges. They just don't work. For us, we see that as a learning experience because then we get to learn OK, well what are we missing? What are the fundamentals that people just aren't getting? What are they not grasping? And we're using that opportunity to really learn from our users and our customers of what do we need to teach them in order to help them really devise a strong challenge? And we've been able to do that. We've been able to take people through trainings and really support them to figure out what this is. And hopefully it'll just continue to sort of extrapolate from there, and their curriculum will get stronger and stronger. This may be too difficult a question, but I am glad to hear Gerald and of some of you say innovation is an overused word. But I feel like it's the holy grail here. I went to school here before that. I was in architecture school. And we were taught anything that was hysteresis was evil, and it all had to be modern, all had to be innovative. Innovation, innovation, innovation. And I feel like we're on a disaster course when we think innovation is going to get us where we need to go. Where are the values? Those briefs are very important, and it's not all about measuring our gross national product by how much suburban farmland we dig up for more suburbs. We're missing it somehow. I don't know if you have any reaction that. But I am sick of the word innovation. Well, just one thought. If you look back, I went to this awesome conference at Virginia Tech and the Smithsonian Lemelson Center put on a few weeks ago that was titled, Can Innovators Be Made? And it looked at the history of entrepreneurship, invention, ideas, innovation, kind of those terms and how they've been used. What's so interesting, if you go back to like the 1700's, 1800's, and you look at how the word innovation was used, those were either like religious heretics or folks that were part of the French Revolution. Innovation was bad. Innovation meant like rebellion. And there's still some of that kind of terminology that's associated with the term today, when you hear innovation. That rebellion is not seen as quite such a bad thing in the terminology of how innovation is used today. And so I struggle too with this notion of innovation is not a solution for anything. And in a lot of ways folks equate ideas and innovation, which is completely false. If you're trying to define innovation as actually an idea realized, then there's a lot of work. And this is where entrepreneurship is largely a bridge between idea and the actual realization of the thing that could be creating change. And so I agree that we get into a lot of it. We're causing the word to have no meaning. And when we really are talking about innovation, or when we are really talking about systemic change, or radicalizing in a particular thing, that's a very specific almost way of looking at that term. That is not the way that people use it broadly now. So I have the same fatigue of the term. But I work in open innovation. My title's Open Innovation. What the hell does that even mean? You now? It's taking innovation and making it open all. That's even more nebulous than the term innovation. Gerald, so we should be wrapping up. Well, can we have one more? Can I just make a 30 second comment? I went to innovation. I'm an engineer, and I've tried to find it. You heard Jenn talk about it. And don't say buy my book, but I go through and talk about what innovation is. And then when you define it like that like, then all of a sudden we can have a conversation. But we can't have a conversation when Ford says, the new innovative F-150. What's that mean? Innovative? It's just like a marketing term. If you could keep it short, then we can deal with it. I was just wondering how the challenges are financially sustainable? Is it through entrance fees or through sponsorship? It's a great question. A few different models. For example, at NASA for some of the challenges that were run through a third party, people would pay $1,000 registration fee in order to get parts. So part of why you did that was to try to cut-- separate the wheat from the chaff. There was a little bit of a barrier to entry in competing to try to gauge seriousness of folks that were competing. Now that was through a third party. So the government Treasury was not taking in a thousand dollars from individual teams. It's a key part of the way that those were run. But some of it depends on who is benefiting from the invention, or the idea that comes out of it, types of barriers to entry that you want to actually create or take down in the process, and some of it is actually-- a lot of it is government appropriated funds. But in some cases, like with Rebuild by Design, Rockefeller Foundation and some others put up money to actually support that $2 million. So no HUD money, other than Scott's salary was spent on the HUD Rebuild by Design Challenge, because it was in partnership with philanthropic partners. That was another $2 million, wasn't it? Your salary? Just for a completely different one. At MIT, how do we get funded? Over 80%, I think it's 86% of the money that comes in to MIT, comes from entrepreneurs. And so they are the most generous people that give. And it's not their first company. And if you just walk around MIT, it's the Stata Center, it's the Koch Center, it's the Porter Building. It's all these people. And they're all entrepreneurs. McGovern Center. They're all entrepreneurs. And so if you can be patient with it, they're extremely generous. Because they realize this is where they learned this capability. This idea that I find horrible now of schools taking a piece of student's companies, so they can make money. I find this just so wrong at a moral level, and so wrong at a financial level. If you allow them to learn how to fish-- it's not going to be the first fish they catch. It's going to be the third fish, data shows. And they'll be very grateful that you taught them how to do that. And if you can be patient, the data is very clear at MIT, they are enormously grateful for that. So XPRIZES are funded by really, really, really, wealthy people or really large companies like Google and the Ansari family and things like that. So they front the money. $10 million, $25 million. Things like that. There is usually a registration fee with XPRIZES because they are usually very technically intensive. And so to operate XPRIZES is actually very, very significant. They can run between $3 million and $7 million to actually operate the challenge itself. With HeroX, we charge a platform fee for anyone who runs a challenge on our platform. But the prizes themselves are sponsored by the customer or client. But we also have the capability and we're supporting companies, individuals to actually crowd fund prizes as well, a la Kickstarter. Because we do believe that just like Kickstarter, people will want to be a part of a solution to help solve problems. And so we want to be able to give people the opportunity to crowd fund the prize dollars as well. And your platform fee is based on the size of the-- On the size of the prize. Something we need to learn. And every vendor has a very different pricing model. So that's another interesting thing in the space. HeroX has a very different pricing model than TopCoder, which is very different than InnoCentive, which is very different than Kaggle and Tongal and GrabCAD. There's dozens of these online platforms that specialize in different kind of communities that you can tap into for different types of problems. It's why it makes them hard to buy, from the government perspective. Because there's not one way that you can price out proposals against one another. Thank you very much everybody. [applause]