How to not f*&k up AI projects in three easy steps


Sorry for the explicit language in the title, but examples, like MD Anderson having to bench Watson in its "eradicate cancer" moon shot, after a 3-year, $62 million dollar effort, make my head explode. 

I regularly tell executives to think of artificial intelligence as the rocket science of a space program.

Rocket science is relatively simple, compared to a space program. AI is the (relatively) easy part of the business problem they are trying to solve with AI. Guaranteed. From almost 30 years dancing in and around AI, including implementing multi-million dollar expert systems at some of the world's largest companies, these are my three steps to help ensure your $62 million or even $6200 does not meet the same fate as the Oncology Expert Advisor.

As the audit report on the project says, "results herein should not be interpreted as an opinion on the scientific basis or functional capabilities of the system in its current state." (Note, they plan to review the technology separately.) In this case, the "rocket science" AI used here may or may not prove out, but we do know from this audit that this "space program" failed. 

First, define and repeat a clear vision. Since this project was named a moon shot, let's take inspiration from the speech that defined moon shot, JFK to congress in 1961 said:

I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the Earth.

Note the big vision AND the specificity. I have met and worked with many NASA engineers. I have been struck by how often the lives of the astronauts came up in discussions about their work. This happened even when that work was on unmanned  tests, or even computer simulations, associated with space travel -- and these conversations were more than thirty years after his declaration. I believe this enduring vision came from the clarity of what JFK presented as the challenge: land a man on the moon and return him safely to Earth.

What I heard most often about the MD Anderson/Watson collaboration was that the goal was to eradicate cancer. Granted I've talked with more NASA engineers than MD Anderson/Watson engineers, but I never heard much more clear vision than that regarding the project. Reading the 2013 press release on the project, I also did not find one.

Taking a non-MD swing at this, I'd propose a vision like, "Eradicate cancer by delivering the most effective treatment recommendations by cancer type." Or, "Eradicate cancer through ubiquitous usage of Oncology Expert Advisor." Those are two ideas of where I would start. I like the second better because of the specificity that people must use the tool, which also implies it must be effective. 

It is easy to put a big vision in place. It is hard to make that vision felt every day by your team. When you have the right vision, people will live that vision in the way you intended. Use JFK's moon shot vision as a benchmark. Talk with your team about your vision. Does it give them the daily aspiration and guidelines needed? If not, revisit. And DO NOT repeat until you've got it. Then repeat, repeat, repeat.

Second, define and track first steps. For years, I've used Dannemiller's awesome "change formula" to understand why a project is not moving forward. This is a picture of it from my current Shinola notebook:

Formula for Change

To explain: Change (delta) happens (=) when dissatisfaction with current situation (frownie) comes together with a vision (cloud) and first steps (steps) to form something greater than resistance (ohm). I don't think anyone will dispute current dissatisfaction with cancer. We covered vision above. So, let's talk about first steps. 

From the audit report, see the Vendor Delivery on Contract Requirements (pages 10-11). While it is a short section, it is clear that the agreed pilots were not done; cancer type target for pilots was switched causing data issues; and invoices were paid without verification of tracking to timelines or requirements. These are all indications that there was not adherence to defined steps towards the vision. 

Continuing with our space program example, Saturn (rockets) and Gemini (manned, orbiting) projects were both steps towards Apollo, Kennedy's moon shot success. Sometimes you have failures in your steps, like Apollo 1. You must learn from those failures to prevent failures moving forward. I wonder if there was an after action review of the cancer-type switch that was at least one cause for the pilots not being started. 

Get your first steps down, and then track them. After you've successfully completed those, define your next first steps. Rinse, repeat until you've reached your vision.

Third, force transparency. It is clear from the audit report that there was a lack of transparency across groups. Procurement and IT guidelines were not followed as examples. The report suggests that this was done intentionally. Do not assume that just because you trust your team not to do this intentionally, they won't walk into a lack of transparency completely honestly. This happens regularly due to organizational silos. AI programs, like space programs, are inherently interdisciplinary. At the very least, you have data, technology and business coming together. Each of these can have multiple groups as stakeholders. You must set-up processes and expectations that people work to bridge the inherent divides between groups.

And, not only is the program vulnerable to lack of transparency, but so is the rocket science. In the words of DARPA:

The effectiveness of these systems is limited by the machine's current inability to explain their decisions and actions to human users.

Most deep learning systems are black boxes. There are efforts to make them share their secrets, as well as hacks (generally computationally intensive) that might work depending on the problem you are solving. There are also solutions -- Yes, like ours at Nara Logics -- that inherently provide transparency. (For us, providing the reasons "why" for every recommendation we make is a founding principle.) You must push your team to force transparency in the algorithm, as much as possible, even if it is via hacks for your proof of concept.

The more you raise demand transparency, the more the AI industry will respond.

This transparency is not only critical for adoption, as DARPA points out, but also for finding the bugs in your data. If you don't know why the algorithm recommended something, you cannot understand why it might be wrong. Like you had to in school, we must expect the algorithms to explain their work. Following from the proposed vision, this transparency would allow doctors and clinicians to give direct feedback as to why a recommendation did not fit. This could suggest new data sources or also indicate areas where current sources need clarification.

Now, you have my critical three tenets for managing your AI project. My exploded head feels better after getting this off to you. In the words of another space hero, here's to your AI projects go SUCCESSFULLY "to infinity and beyond!"