Today, organizations and enterprises of most every kind must be able to tap into what is now an essential, yet challenging resource: big data. Composed of internally stored organizational information such as customer and sales information, transactional data, research, as well as external open source information and social media, this big data is largely unstructured and in a state of constant growth. It is also mostly text. Natural language processing (NLP) of big data is the next great opportunity.
What is big data?
No longer just a buzzword, the phrase “big data” describes the growing volume of structured and unstructured, multi-source information that is too large for traditional applications to handle. In terms of its usefulness, the 2013 book “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” by Viktor Mayer-Schönberger and Kenneth Cukier refers to big data as “the ability of society to harness massive amounts of information in novel ways to produce useful insights or goods or services of significant value.”
What is NLP?
Natural language processing (NLP) is a form of artificial intelligence that helps machines “read” text by simulating the human ability to understand language. NLP techniques incorporate a variety of methods, including linguistics, semantics, statistics and machine learning to extract entities, relationships and understand context, which enables an understanding of what’s being said or written, in a comprehensive way. Rather than understanding single words or combinations of them, NLP helps computers understand sentences as they are spoken or written by a human. It uses a number of methodologies to decipher ambiguities in language, including automatic summarization, part-of-speech tagging, disambiguation, entity extraction and relations extraction, as well as disambiguation and natural language understanding and recognition.
What problems can natural language processing for big data solve?
Regardless of the sector, every business today relies on large volumes of text information. For example, a law firm works with large amounts of research, past and ongoing legal transaction documents, notes, email correspondence as well as large volumes of governmental and specialized reference information. A pharmaceutical company will have large volumes of clinical trial information and data, doctor notes, patient information and data, patent and regulatory information as well as the latest research on competitors.
Because these types of information are largely made up of language, natural language processing for big data presents an opportunity to take advantage of what is contained in especially large and growing stores of content to reveal patterns, connections and trends across disparate sources of data.
Interactions: Today, natural language processing technologies are already at work in a variety of commonly used interactive applications such as smartphone assistants like Apple’s Siri, in online banking and retail self-service tools and in some automatic translation programs. Users ask questions in everyday language and receive immediate, accurate answers. It’s a win-win for both customers, who can easily communicate with companies they do business with whenever and wherever they want, and for companies who increasingly realize savings by reducing the number of calls handled by traditional live assistance.
Business Intelligence: Natural language processing for big data can be leveraged to automatically find relevant information and/or summarize the content of documents in large volumes of information for collective insight. Users are no longer limited by having to choose or know the “right” keywords to retrieve what they’re looking for but can interact with the content via search using queries in their own words. Faster, more thorough access to information speeds up all downstream processes that depend on timely information and enable its use for real time, actionable business intelligence.
Sentiment analysis: With an increasingly online customer base, social channels are a rich, if noisy source of invaluable information. Using natural language processing for sentiment analysis, organizations can understand what is being said about their brand and products, as well as “how” it’s being talked about—how users feels about a service, product or concept/idea. This is a powerful way to discover information about the market and about current and potential customers (opinions, but also information about customer habits, preferences and needs/wants, as well as demographic information) that would otherwise remain out of reach. This information can then be applied to product development, business intelligence and market research.
If estimates by IDC come true, by 2020, we’ll be looking at around 44 trillion gigabytes of digital knowledge worldwide (an IDC Digital Universe Study reports that by 2020, for every human in the world, approximately 1.7 megabytes of new information will be created every second; that’s around 44 trillion gigabytes). Fourty four trillion gigabytes is a lot of potential. No matter where you apply it, natural language processing for big data will be an essential feature to build into your analysis pipeline in order to capture the value of this information for insight, reduced costs and increased productivity.
