Natural Language Processing (NLP) is a field of Artificial Intelligence which enables computers to analyze and understand the human language. It was formulated to to build software that generates and understand languages so that a user perform an action without explicitly doing it.
If one had a database of all the eateries in India then for them to find restaurants serving Thai cuisine in New York, They’d typically write a SQL query like this:
SELECT restaurant_name, restaurant_address FROM restaurants WHERE city = ‘New York’ AND food_type = ‘Thai’
On to the Natural way:
Where can I find some Thai food in New York?
In other words, letting the algorithm decide and mine information based on free text input provided by the user. The more widely used practical example for NLP is Conversational AI, i.e. Siri, Google Assistant, Cortana, and 10 billion other variations to them.
NLP use-cases are not limited to just extracting information but a great variety of other tasks ranging from Text Categorization to Question Answering.
It is the task of bucket-ing text into categories within an assumption that texts in similar categories follow the same distribution of words. It is widely used in building systems like Sentiment analysis, News article categorization, Customer reviews segmentation and many more.
It refers to the technique of shortening long pieces of text. The intention is to create a coherent and fluent summary having only the main points outlined in the document.
It refers to fully automated software that can translate source content into target languages. Humans may use it to help them render text and speech into another language, or the software may operate without human intervention. example, Google Translate, Bing Translate, Yandex, etc
A natural language search is a search using regular spoken language, such as English. Using this type of search you can ask the database a question or you can type in a sentence that describes the information you are looking for. The database then uses a programmed logic to determine the keywords in the sentence by their position in the sentence. example, Google Search, Duck Duck Go.
It is a process to automatically identify topics present in a text object and to derive hidden patterns exhibited by a text corpus. It comprises of plenty of statistical modeling techniques like Latent Dirichlet Allocation, Latent Semantic Indexing and simpler rudimentary techniques like word co-location analysis, counts, word clouds, etc.
Question answering (QA) is a computer science discipline within the fields of information retrieval and NLP, which is concerned with building systems that automatically answer questions posed by humans in a natural language. Example, Alexa, Google Assistant, etc
While not cut and dry, there are 3 main groups of approaches to solving NLP tasks.
Rule-based approaches are the oldest approaches to NLP. Why are they still used, you might ask?
It’s because they are tried and true, and have been proven to work well. Rules applied to text can offer a lot of insight: think of what you can learn about arbitrary text by finding what words are nouns, or what verbs end in -ing, or whether a pattern recognizable as Python code can be identified.
Regular expressions and context free grammars are textbook examples of rule-based approaches to NLP.
"Traditional" Machine Learning
It approaches include probabilistic modeling, likelihood maximization, and linear classifiers, Traditional machine learning approaches are characterized by:
Training Data - in this case, a corpus with markup
Feature Engineering - word type, surrounding words, capitalized, plural, etc.
Training a model on parameters, followed by fitting on test data (typical of machine learning systems in general)
Inference (applying model to test data) characterized by finding most probable words, next word, best category, etc.
This is similar to “traditional” machine learning, but with a few differences:
Feature Engineering is generally skipped, as networks will “learn” important features (this is generally one of the claimed big benefits of using neural networks for NLP)
Instead, streams of raw parameters (“words” – actually vector representations of words) without engineered features, are fed into neural networks very large training corpus
Specific neural networks of use in NLP include recurrent neural networks (RNNs) and convolutional neural networks (CNNs).
Why is NLP so hard?
NLP is hard for two primary reasons: humans don’t always express intent through semantically accurate language, and there are numerous ambiguities in language. Some examples include:
- Semantics “Gabe invited me to his medical school ball”. What is “ball” in this context?
- Morphology (parts of the word that can be deconstructed to create different meanings).
- Ambiguity of intent: “I just got back from New York”. What do they want?
- Situational ambiguity: “Elaina was found by the river head”. Could be by the head of the river (place) or the executive of the river (person)
- Unable to deduce meaning of unknown words from context like humans can
- Disambiguation – “jaguar” can refer to a car or to an animal
These difficulties arise even when a machine is trying to understand a perfectly written and expressed piece of text. Most of the time, though, the text that the machines get is imperfect – riddled with typos or slang or unclear intentions.
These issues are compounded through voice – in addition to the above difficulties, the machine also has to parse phonetics and phonology, such as determining whether someone is saying “I scream” or “Ice cream”. It’s no wonder that smart speakers still mishear, misinterpret, or overhear and take other conversations as commands.
Natural Language Processing can be daunting and overwhelming but certainly would serve as a huge asset to your Machine Learning workflows, interested in understanding how this may help your organization? Shoot a note out to [email protected]