From the course: Introduction to NLP and LLMs: Principles and Practical Applications
Join today to access over 24,400 courses taught by industry experts.
Why process text data?
From the course: Introduction to NLP and LLMs: Principles and Practical Applications
Why process text data?
- [Instructor] Why process text data? Well, since computers cannot directly understand and process human language in its raw form, we need to pre-process the text data to make it suitable for analysis by machines. Several common techniques are used to prepare text for analysis. For example, their first step is tokenization, which means breaking down the text into individual words or subwords called tokens. Next is removing stop words, which means eliminating common words like the, A, is, which don't usually carry any significant meaning. Lowercasing just means converting all text to lowercase to standardize the data. Stemming or lemmatization means reducing words to their root form. For example, changing the word running to run. Part-of-speech tagging is simply identifying the grammatical role of each word in a sentence. For example, is it a noun, verb, or adjective? Entity extraction means identifying and extracting specific entities like names, locations, organizations from a…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.
