NLP Named Entity Recognition
This project focuses on building and evaluating Named Entity Recognition (NER) models using a combination of traditional sequence models and modern transformer-based architectures. The aim was to identify and classify domain-specific entities from text with high accuracy, while comparing different embedding strategies and learning frameworks.
The project explores how contextual embeddings and sequence modelling techniques improve entity recognition performance across complex textual data.
Problem Statement
Named Entity Recognition is a core task in Natural Language Processing, yet it remains challenging due to contextual ambiguity, domain-specific terminology, and variations in linguistic structure. Traditional models often struggle to generalise across unseen data, while deep learning approaches require careful architecture and loss design to perform effectively.
This project addresses the challenge of accurately identifying named entities by evaluating multiple modelling approaches and understanding the trade-offs between classical NLP pipelines and transformer-based models.
Approach
The approach involved designing and training multiple NER models using different representation and learning strategies:
Implemented BiLSTM-CRF models for sequence labelling with contextual word representations.
Evaluated fastText, Flair, BERT, and BioBERT embeddings to capture semantic and contextual information.
Applied Conditional Random Fields (CRF) to model label dependencies across token sequences.
Optimised training using AdamW and incorporated KL-Divergence loss for improved distribution alignment.
Compared model performance across architectures to analyse robustness and generalisation.
Results
The transformer-based models consistently outperformed traditional embeddings, demonstrating stronger contextual understanding and improved entity boundary detection. In particular:
BERT and BioBERT achieved higher precision and recall across entity classes.
BiLSTM-CRF models benefited significantly from contextual embeddings such as Flair.
Error analysis highlighted improved handling of long-range dependencies and rare entity patterns in transformer-based approaches.
The results confirm that contextual language models substantially enhance NER performance compared to static embeddings.
Tech Stack
Python · fastText · BERT · BioBERT · Flair · BiLSTM-CRF · Conditional Random Fields (CRF) · AdamW · KL-Divergence Loss · Named Entity Recognition · NLP