What is Named Entity Recognition (NER), and what makes it a harder problem than simple keyword matching?
NLPLevel: FoundationNatural language processing
What is Named Entity Recognition (NER), and what makes it a harder problem than simple keyword matching?
Hints
- Separate text representation, modeling and evaluation.
- Check ambiguity, rare terms and training versus target text distributions.
Answer
NER identifies and classifies spans of text into predefined categories (person, organization, location, date, etc.). It's harder than keyword matching because entity mentions are context-dependent and ambiguous — "Washington" could be a person, a state, or a city depending on context, and entity boundaries aren't always obvious (e.g., "Bank of America" is one entity, not two). Modern NER models use sequence labeling (e.g., BIO tagging with a Transformer-based encoder) that considers the surrounding context of each token, rather than matching against a static list of known entity names, which fails on new/unseen entities and ambiguous cases.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT