Natural Language Processing
Language is meant for communicating about the world. The fields of AI attempts to make machines understand human language, both in terms of speech and written form. The largest part of human linguistic communication occurs as speech. Natural language processing (NLP) is the process of computer analysis of input provided in a human language (natural language), and conversion of this input into a useful form of representation.
Entire language processing problem in two tasks:
- Processing written text: Using lexical, syntactic, and semantic knowledge of the language as well as required real world information
- Processing spoken language: All information included in (i) plus additional knowledge about phonology as well as enough added information to handle the further ambuiguity that arise in speech (eg.noise)
Natural Language Understanding (NLU)
- NLU includes understanding the language, generating meaningful sentences to respond to queries a well as other tasks such as multilingual translation.
Problems Related to NLP
There are many problems. The major problems are:
- Natural languages are incomplete description of the information
e.g. Some dogs are outside.
(i) Some dogs are on the lawn
(ii) Three dogs are on the lawn
2. The same expressions means different things in different contexts
E.g. Where’s the water? (In a chemistry lab, it must be pure)
Where’s the water? (When you are thirsty, it must be potable)
3. No natural language program can be complete
Because new words, expressions, and meanings can be generated quite freely. E.g. I’ll fax it to you
4. There are lot of ways to say the same thing.
E.g. Mary was born on October 11.
Mary’s birthday is October 11.
Levels of analysis for natural language
Language is a complicated phenomenon . To manage this complexity, linguists have defined different level of analysis for natural language.
- Prosody:
- Deal with rhythm and intonation of language. This level of analysis is difficult to formalize and often neglected
2. Morphology:
- Examines the sounds that are combined to form language; this branch of linguistics is important for computerized speech recognition and generation.
3. Morphology:
- Concerned with the components (morphemes) that make up words.
- There include the rules governing the formation of words such as the effect of prefixes (un-,non-, anti-,etc) and suffixes (-ing, -ly, etc) that modify the meaning of root words
- Morphological analysis is important in determining the rate of a world in a sentence including its tense, number, and parts of speech
4. Syntax:
- Studies the rules for combining words into legal phrases and sentences and use of those rules to parse and generate sentences.
5. Semantics:
- Considers the meaning of words, phrases and sentences and the ways in which meaning is conveyed in natural language expressions.
6. Pragmatics:
- is the study of ways in which language is used and its effects on the listener
7. World Knowledge:
- includes knowledge of the physical world i.e. the domain in concern.
Steps In Natural Language Processing
- Morphological Analysis:
- individual words are analyzed and non-word tokens such as punctuation, are separated from the words.
2. Syntactic Analysis:
- Linear sequence of words is transferred into structures that show how the owrds relate to each other. Some word sequences may be rejected if they violate the language’s rules for new words may be combined.
3. Semantic Analysis:
- structures are created by the semantic analyzer are assigned meanings.
- syntactically valid sentence may be semantically invalid. Eg. Tiger eat grass.
4. Discourse Integration:
- The meaning of an individual sentence may depend on the sentences that precede it and may influence the meanings of the sentence that follow it. Eg. John wanted it.
5. Pragmatic Analysis:
- The structure representing what was said is reinterpreted to determine what was actually meant. Eg. Do you know what time is it?
Applications of NLP
- Machine Translation – translation between two natural languages
- Information retrieval – web search
- Query Answering/Dialogue – natural language interface with a database system or a dialogue system.
- Report generation – Generation of reports such as weather reports
- Some small application – Grammar checking, spell checking, etc.