“Watch out for arguments about future technology that is magical. Such an argument can never be refuted. It is a faith-based argument, not a scientific argument.”
— Rodney Brooks, Former Director AI and Computer Science Laboratory, MIT
“Show me the money!”
— Jerry Maguire
I am here to propose a radical new benchmark for AI: is it worth more than it costs? This is a respected and time-honored consideration for every business buying decision. And right now, there is no particular reason AI should get a pass on it. I also think this would introduce a bit more pragmatism around what is basically…get ready…software.
AI, at least in its current incarnation, is clearly in the “late hype” phase of its adoption cycle. To be fair, many technologies go through a period of inflated expectations where normal, sober business practices are suspended. And this hype cycle tends to be further exaggerated when the technology in question meets a predictable set of criteria. AI demonstrates all of them on steroids:
- Broad descriptions of profound capabilities that prompt “fear of missing out.”
AI hits some elemental nerve in its claim to replicate — or replace — capabilities that are uniquely human. This seems to generate a (completely misplaced) belief that AI offers the potential for “hands-free,” generic and infinitely scalable solutions to the challenges faced by business. - Claims of even more transformative impact just over the horizon.
There is a constant stream of innovation in AI (e.g., machine/deep/transfer/reinforcement learning, convolutional/recurrent neural networks, transformer language models, etc.). Each new development surfaces new claims about imminent revolutionary impact. The fact that a disturbing number of these “breakthroughs” are non-reproducible or spawn a host of other problems tends to get lost in the noise. - Easily accessible “free” approaches to the technology.
Heck, we can just do this ourselves! Many of the recent approaches to AI emerged from research by Google, Facebook and other big players and were released as “open source.” Not surprisingly, many businesses saw this as “free” and succumbed to the siren’s call of “DAIY” or “Do AI Yourself.”¹ Businesses do need to understand AI capabilities and how to integrate them effectively. I would be much more cautious about building unique, untested AI solutions in-house, as it is really easy to get in over your head, particularly if you are taking on some of the larger challenges such as core language understanding and object recognition (see IBM and Uber below). - Complexity that makes the technology opaque to business decision makers.
AI is a broad category that encompasses many different approaches, and it takes real work to understand how and where each can add value. For starters, AI is not synonymous with machine learning. And to the broader point about benchmarks, many measures of AI performance are — to put it mildly — pretty artificial.²
When Hype Meets Reality
All of this has fed the reinforcement cycle of media hype — “The Robots Are Coming for Your Job” — And not surprisingly, much of the early money invested by businesses in AI has essentially been shoveled into a furnace.
Ask IBM, which put its Watson Health business on the “best offer” block, or Uber, which sold its self-driving car unit after spending a cool $2.5BN. Think about that for a second — IBM and Uber. IBM’s business is technology. Uber built a breakthrough business model based meaningfully on deploying advanced predictive analytics at scale. When those two companies waste a lot of money on AI, it should serve as a sobering wake-up call.
These costly forays into admittedly big challenges are being repeated at a smaller scale across the AI landscape. An MITSloan study estimated about 90% of AI projects fail to deliver expected ROI. And while many of the early adopters are the most capable at evaluating the technology and putting it to good use, the move up the curve towards mass adoption has proven messy and expensive.
A Simple Benchmark
This is where businesses have to impose the new AI benchmark — is it worth more than it costs? These considerations may prove useful in answering that question:
- Define the job to be done and the value you want to create.
Hint: it should be pretty clear how you will make or save money. An awful lot of AI work is charitably described as “experimental.” This is fine if you can afford it, but it’s generally not the goal for most IT projects. Know the problem or the opportunity clearly enough to put a real dollar figure around the upside you can realize with a successful implementation. - Determine the data involved and why the AI approach you have chosen is suited to the task.
Data is critical in AI, but up to 80% of your typical AI project is often consumed by data-wrangling of various sorts, says Cognilytica. Training a machine learning system requires large numbers of carefully labeled examples, and those labels are usually applied by humans. Many AI approaches work reasonably well when the data is structured and abundant, but others are best suited to unstructured data like language and work well even with small amounts. That’s our focus. - Scope the work necessary to get to scale and keep it operational.
You will need to do work, so make sure you understand what that work involves. No one thinks twice about the effort put into customizing a web application or an ERP solution, but part of the inflated expectations problem is many folks seem to think that with AI, you can “set it and forget it.” Big mistake. Ask detailed questions about the amount and type of work required for the project at every stage. - Think through the full range of tradeoffs and costs.
AI approaches involve many tradeoffs including cost, bias, explainability, flexibility, reusability, privacy, and even carbon footprint. Make sure you run through that checklist very carefully and, by all means, include the entire team (i.e., IT, data, business line) as you think through the full implementation of an AI-driven capability. - Assess the flexibility and “staying power” of your AI approach.
Have a point of view around the core set of AI technologies you think are most relevant to your business. Yours is likely to be pretty different than that of a company in a different industry. Then, make sure you have an approach that allows you to integrate what actually works and avoid the dreaded technical debt.³ We have done exactly that in building our natural language understanding platform, so we are definitely following our own advice. - Finally…ask if it has worked before in a similar use case.
Pretty standard question for any software.
The computational advances of AI are very exciting and will likely change our lives over time in completely unforeseen ways. But you can say the same thing about the advances in solar-generated power and battery storage that are taking place really quickly and that have the potential to…err…save the planet.
The point is, businesses need to solve problems now, and they want results (not experiments or vague promises of a powerful future). This, admittedly, is a generalization, but it doesn’t hurt to think about AI as software. So before spending money on AI, understand why it is worth more than it costs. At expert.ai, we are definitely up for a conversation about our performance on that AI benchmark.
Footnotes:
- Clemens Mewald of Databricks called this the “IKEA effect” — it’s better because I built it. He further noted that ML models basically fall under the broad category of “data-driven applications.”
- Examples include most of the language understanding benchmarks. By many of these measures, a number of AI language understanding models perform at a level equal to that of humans. This is nonsense.
- A Google research paper in 2014 called machine learning the “high interest credit card of technical debt.” I’m not sure that is really true anymore for some ML use cases, but I had to get that in here.
