List of Figures xiii List of Tables xv About the Author xvii Preface xix Acknowledgements xxi 1 Foundations of Natural Language Processing 1 1.1 History of NLP 3 1.2 Approaches to NLP 5 1.3 Understanding NLP through NLU and NLG: Examples and Case Studies 9 1.3.1 Practical Case Studies 9 1.4 NLP Pipeline 10 1.5 NLP’s Transformative Impact on Business and Society 12 2 Navigating the Data Landscape for NLP 15 2.1 Types of Data in NLP 16 2.2 Data Acquisition 17 2.3 Challenges in NLP Data Acquisition and Management 21 2.4 Data Quality Check in NLP 22 2.5 Ethical Considerations in NLP Data Management 25 3 Fundamental Text Processing 31 3.1 Text Cleaning 32 3.2 Sentence Splitting 34 3.3 Tokenization 36 3.4 Lemmatization and Stemming 44 3.5 StopWord Removal 48 3.6 Part-of-Speech Tagging 49 3.7 Parsing and Syntactic Analysis 50 3.8 Tools and Libraries for Text Processing 56 4 Linguistic Features in NLP 63 4.1 Levels of Linguistic Analysis 64 4.2 Features in NLP 73 4.3 Vector Space Representation in NLP 75 4.4 Semantic Features in NLP 81 4.5 Feature Generation in NLP: Manual versus Automatic Approaches 89 5 Computational and Cognitive Approaches in Natural Language Processing 95 5.1 Machine Learning for NLP 97 5.2 Memory and Recall Models 100 5.3 Attention Mechanisms 105 5.4 Human-Like Reasoning 112 5.5 Transfer Learning in NLP 114 5.6 Learning with Minimal Examples 122 5.7 Neuro-Symbolic Approaches 123 6 Fundamental Language Processing Techniques 129 6.1 Topic Modelling and Subject Identification 129 6.2 Named Entity Recognition 136 6.3 Text Coherence and Cohesion 144 6.4 Stylistic Analysis 151 6.5 Semantic Role Labelling 154 7 Natural Language Processing for Affective, Psychological, and Content Analysis 159 7.1 Sentiment Analysis: Dissecting Text for Opinion Mining 159 7.2 Emotion Recognition: Beyond Polarity 168 7.3 Irony and Sarcasm Detection: Between the Lines 175 7.4 Humor Identification in Text: Tapping into Textual Tickle 180 7.5 Psychometric NLP 184 7.6 Learning Disabilities Detection 190 7.7 Textual Indicators of Distress: Addressing Depression, Anxiety, and Beyond 194 7.8 Digital Content Moderation using NLP 198 8 Multilingual Natural Language Processing 223 8.1 Translation and Transliteration 223 8.2 Cross-Lingual Models and Embeddings 228 8.3 Low-Resource Language Processing 235 8.4 Cultural Nuance and Idiom Recognition in Natural Language Processing 240 9 Domain-Specific Natural Language Processing 243 9.1 Healthcare Natural Language Processing 243 9.2 Legal Natural Language Processing 250 9.3 Finance Natural Language Processing 255 9.4 NLP in Education 262 10 Measuring Success in Natural Language Processing Evaluation and Metrics 269 10.1 Intrinsic versus Extrinsic Evaluation Techniques 269 10.2 Extrinsic Evaluation Techniques 270 10.3 Metrics for Text Classification 272 10.4 Evaluating Machine Translation and Text Summarization 276 10.5 Metrics for Question-Answering and Conversational AI 278 10.5.1 Evaluation in Ranking and Information Retrieval 279 10.6 Metrics for Text-Based Forecasting and Prediction 280 Knowledge Checkpoint Answers 285 A Text Representation Techniques: A Unified Overview 305 B Step-by-Step Guide to NLP Processing on E-Commerce Customer Feedback 307 C Harnessing Python Libraries for NLP 311 Further Reading 313 Index 315

Thorough review of foundational concepts and advanced techniques in natural language processing (NLP) and its impact across sectors Supported by examples and case studies throughout, Language Intelligence provides an in-depth exploration of the latest advancements in natural language processing (NLP), offering a unique blend of insight on theoretical foundations, practical applications, and future directions in the field. Comprised of 10 chapters, this book provides a thorough understanding of both foundational concepts and advanced techniques, starting with an overview of the historical development of NLP and essential mechanisms of Natural Language Understanding (NLU) and Natural Language Generation (NLG). It delves into the data landscape crucial for NLP, emphasizing ethical considerations, and equips readers with fundamental text processing techniques. The book also discusses linguistic features central to NLP and explores computational and cognitive approaches that enrich the field’s advancement. Practical applications and advanced processing techniques across various sectors like healthcare, legal, finance, and education are showcased, along with a critical examination of NLP metrics and methods for evaluation. The appendices offer detailed explorations of text representation methods, advanced applications, and Python’s NLP capabilities, aiming to inform, inspire, and ignite a passion for NLP in the ever-expanding digital universe. Written by a highly qualified academic with significant research experience in the field, Language Intelligence covers topics including: Fundamental text processing, covering text cleaning, sentence splitting, tokenization, lemmatization and stemming, stop-word removal, part-of-speech tagging, and parsing and syntactic analysisComputational and cognitive approaches, covering human-like reasoning, transfer learning, and learning with minimal examplesAffective, psychological, and content analysis, covering sentiment analysis, emotion recognition, irony, humour, and sarcasm detection, and indicators of distressMultilingual natural language processing, covering translation and transliteration, cross-lingual models and embeddings, low-resource language processing, and cultural nuance and idiom recognition Language Intelligence is an ideal reference for professionals across sectors and graduate students in related programs of study who have a foundational understanding of computer science, linguistics, and artificial intelligence looking to delve deeper into the intricacies of NLP.