Why Picking the Right NLP Technology Matters
We all know the old adage, “When all you have is a hammer, everything looks like a nail.” But not everything is a nail, especially when it comes to documents and content.
When we work on a document, we must understand its format and what it is about. If I start working at the DMV, for example, I have to quickly understand their forms and what problems they address, the questions drivers have and the most appropriate answers to those questions. If I work for the Department of Agriculture, then I need a completely different set of tools.
This simple concept may be obvious to us, but it seems to escape us in the case of Natural Language Processing. This industry is very diverse; it’s populated with many vendors who apply completely different methodologies and offer very distinct functionalities. Still, when facing use cases that involve documents in any shape or form, organizations look at NLP vendors as if they all provide comparable solutions. Documents are nails, NLP is the hammer.
The truth is, NLP is not one thing. Documents provide value to each company only if they’re leveraged in very specific ways, and yes, you need Natural Language Processing, but not because it’s a hammer. It is, in fact, a toolbox.
The history of how this technology evolved can explain why we have this diversity of tools today. Without getting into the details, we essentially started with a very simplistic approach, and because that worked, the demand for such tools grew. As a result, many new players joined the party. In a competitive environment, vendors try to have differentiating factors; in the NLP industry, some decided to make the original technology even better (developing their instruments in a horizontal way), some decided to specialize their products to become the best in a specific industry (they moved vertically) and others decided to use original approaches that would present a more appealing set of advantages or a shorter list of drawbacks.
Today, the industry as a whole can be seen as a toolbox. Some companies offer a hammer, some have screwdrivers. You can meet those who offer the best hammer, those who sell scalpels (which you only need if you’re a doctor), and of course those who have hammers and shovels (which might interest you if you think that you need a hammer today, but in the future you may need something else). None of us would buy a scalpel if they have to put a nail in the wall.
Oddly enough, this is what happens, more often than not, when organizations look for an NLP solution. But whose fault is it? I don’t believe we should expect companies to understand NLP more than I understand the pill my doctor prescribes for when I have back pain.
Most NLP developers like to present their offer as a broad range of solutions. Whenever they approach (or are approached by) a prospective client, they will assess the problem in front of them and consequently propose the tools they have available, sometimes, forcing those tools into the picture. We can, after all, put a nail in the wall using a shovel. Can we blame them? Private companies are in business for one reason only: to make money. They are not known for leaving the room when there is money on the table. But this behavior is strongly affecting the NLP world, which is a very young industry that is still surrounded by a lot of ignorance. Clients don’t get the results they’d like (or that were promised), therefore they assume the technology is not ready for prime time.
To frame this differently: if a technology expert isn’t honest about what their product can achieve, then their prospective clients are unable to have a complete understanding of what makes every vendor unique, and they just aim straight at the lowest price tag. They don’t even wonder why there may be such a large price gap between two solutions; to them, NLP is all the same. This is the outcome we often experience when a technology company tries to make a sale at all costs. And, as many things in nature, when people think only about themselves they fail the environment and, ultimately, compromise their own future.
This industry is not old enough to have standardized priorities or that can offer brokers that interface between vendors and clients; this problem can only be solved by ourselves, the NLP technology vendors. Although this might sound naïve, I think that more honesty would do us good. Admitting that a problem is out of our reach would result in happier clients that end up working with valuable solutions, and, as a direct consequence, a market growing faster and a sizable number of new opportunities for everyone. This would enable more competition, but also more collaboration and knowledge distribution, therefore pushing everyone to enhance their technology and broaden their offering.
There might be market consolidation here and there, but this wouldn’t be a zero-sum game; Natural Language Processing can be applied to every single problem that deals with communication, and it’s still just scratching the surface of the revolution to come. In a few years, there will be no text on the planet that doesn’t go through some form of processing, no marketing strategy that doesn’t come from advanced analytics, no bank request that is approved without the help of text mining tools, and so on. We don’t want to slow down this digital transformation process. Instead, we would all benefit from a more solid market that understands our technology and believes in its power.
Filiberto Emanuele
Chief Partnership Officer, expert.ai
