Natural Language Processing
93 flashcards by MentisLibrary
How machines handle text: preprocessing and tokenization, representations from bag of words to embeddings, classification and sequence labelling, transformers and pretraining, language models, and how any of it is evaluated.
- What is natural language processing?
- The field concerned with getting computers to work with human language: understanding it, generating it, and extracting structure from it.
- What is tokenization?
- Splitting a string into the units a model will treat as atomic, usually words or subwords.
- Why is splitting on whitespace not enough?
- Punctuation sticks to words, contractions hold two words, and many languages do not put spaces between words at all.
- What is subword tokenization?
- Splitting rare words into smaller pieces that are in the vocabulary, so any string can be represented without an unknown token.
- How does byte pair encoding build a vocabulary?
- It starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary reaches the target size.
- What is an out-of-vocabulary word?
- A word the model's vocabulary does not contain, traditionally replaced by an unknown token.
- Why does token count matter when using a transformer?
- Models have a maximum context length, and cost usually scales with the number of tokens processed.
- What does text normalisation involve?
- Putting text into a canonical form: case folding, Unicode normalisation, whitespace collapsing and often accent or punctuation handling.
- What is the difference between stemming and lemmatization?
- Stemming chops affixes with rules and may produce non-words; lemmatization maps a word to its dictionary form using a vocabulary and its part of speech.
- What are stopwords, and when should you keep them?
- Very frequent function words such as the, of and is. Keep them whenever word order or negation carries meaning.
- Why is sentence segmentation harder than splitting on a full stop?
- Because a full stop also ends abbreviations, decimals and initials.
- What is Zipf's law, and what does it imply for NLP?
- Word frequency is roughly inversely proportional to frequency rank: a few words are extremely common and a long tail occurs once or twice.
- What is the difference between a type and a token?
- A token is one occurrence of a word in text; a type is one distinct word form in the vocabulary.
- When is edit distance the right tool?
- When you need to measure how many single-character insertions, deletions or substitutions separate two strings.
- What is a bag-of-words representation?
- A vector of word counts, one dimension per vocabulary term, discarding word order entirely.
- What is an n-gram?
- A contiguous sequence of n tokens, used to give a bag-of-words model a little local order.
- How is TF-IDF computed?
- Term frequency in the document, multiplied by the inverse document frequency across the corpus.
- Why is cosine similarity used to compare documents?
- It compares the angle between vectors rather than their magnitude, so a long document and a short one on the same topic still match.
- What is wrong with one-hot word vectors?
- Every word is equally distant from every other, so the representation encodes no similarity at all.
- What is a word embedding?
- A dense, low-dimensional vector for a word, learned so that words used in similar contexts land near each other.
- What is the distributional hypothesis?
- Words that occur in similar contexts tend to have similar meanings, which is the assumption every embedding method rests on.
- How does the skip-gram model learn?
- It trains a model to predict the surrounding context words from a centre word, and keeps the learned input vectors as the embeddings.
- What is negative sampling for?
- It replaces the expensive softmax over the whole vocabulary with a handful of binary decisions against randomly drawn non-context words.
- How does GloVe differ from word2vec?
- GloVe factorises a global word co-occurrence matrix, while word2vec learns from local context windows one at a time.
- What is a word analogy in embedding space?
- The observation that vector offsets can encode relations, so a relation learned for one pair partly transfers to another.
- What is the difference between static and contextual embeddings?
- A static embedding gives a word one vector regardless of use; a contextual embedding produces a different vector for each occurrence.
- Why do word embeddings reproduce social bias?
- They learn the statistics of the text they were trained on, including the stereotyped associations present in it.
- What does a language model do?
- Assigns a probability to a sequence of tokens, usually by predicting each token from those before it.
- How does an n-gram language model estimate probabilities?
- It assumes a token depends only on the previous n minus 1 tokens, and estimates that from corpus counts.
- Why does an n-gram model need smoothing?
- Because any n-gram unseen in training gets probability zero, which makes the probability of the whole sentence zero.