Natural language processing (NLP) does not follow one universal pipeline. A system might match rules, retrieve documents, classify text with a trained model, generate text with a transformer, or combine several of these approaches. What happens between input and output depends on the task and the system design.
This lesson explains the mechanics behind representative NLP systems, separates training from runtime use, and shows what tokens, token IDs, embeddings, and contextual representations each contribute.
How NLP Works in One Minute
NLP systems receive language and produce a useful result: a matched rule, a ranked document, a label, extracted information, a translation, or generated text. They do not all reach that result in the same way.
- Rules and retrieval: match patterns or find relevant information in a collection.
- Classical machine learning: convert text into task-relevant numerical features, then use a trained model to predict a label or score.
- Transformer-based NLP: tokenize input, map token IDs to vectors, build context-sensitive representations inside transformer layers, and use those representations to classify, retrieve, translate, or generate.
Real products often combine these designs. A support assistant, for example, might apply safety rules, retrieve approved documentation, and use a language model to draft an answer.
There Is No Single NLP Pipeline
The phrase “NLP pipeline” describes the ordered components of a particular system—not a mandatory sequence for the entire field. Tokenization and embeddings are central to many model-based systems, but a regular-expression rule or exact-match lookup may need neither.
Preprocessing is also task- and model-dependent. Some systems normalize whitespace, casing, or characters. Others preserve capitalization, punctuation, spelling, or emoji because those details carry useful information. A pretrained model should generally receive the form of input expected by its associated tokenizer and training setup; the Hugging Face tokenizer documentation shows how tokenizers produce model inputs such as input IDs and attention masks.
Training and Inference Are Different Stages
Beginners often picture a model learning whenever a user submits text. Usually, development happens first and runtime use happens later.
| Stage | Typical activities |
|---|---|
| Development and training | Define the task, collect or select data, choose representations and an approach, train or adapt the system, validate it, and test it before deployment. |
| Inference or runtime | Receive new language input, apply required preprocessing or tokenization, run existing rules, retrieval, or a trained model, then return and possibly post-process the result. |
A deployed system may later be updated or retrained, but it normally does not retrain from scratch for every request.
Three Common NLP System Designs
1. Rules and Retrieval
A rules-based system follows explicit instructions. An email filter might flag a message when a known pattern appears. A retrieval system processes a query, searches an index or document collection, and ranks matching results. Neither design requires a language model.
Example flow: language input → optional normalization or query processing → pattern matching or retrieval → result.
2. Classical Machine-Learning NLP
A focused classifier can represent text with features such as word or character counts, n-grams, or TF-IDF values. During development, a model learns statistical relationships between those features and labeled examples. At runtime, the same feature process is applied to new text and the trained model returns a label or score.
Example flow: new message → task-appropriate preprocessing → numerical features → trained classifier → spam probability.
3. Transformer-Based NLP
Transformers are attention-based neural-network architectures widely used in modern NLP. The architecture was introduced in the 2017 paper Attention Is All You Need. They support classification, retrieval, translation, summarization, and generation, but they are not synonymous with NLP.
A typical transformer system uses a model-specific tokenizer, converts tokens to integer IDs and other inputs, maps those IDs to learned vectors, and processes them through transformer layers. Attention helps the model combine information from relevant positions. The resulting contextual representations feed a task-specific component or a decoder that produces the output.
Example flow: text → model-specific tokenizer → token IDs → initial embeddings → transformer layers and attention → contextual representations → task head or decoder → output.
Tokens, Token IDs, Embeddings, and Contextual Representations
These terms describe different things:
- Tokens are the text units selected by a tokenizer. They may be words, parts of words, punctuation, or other pieces.
- Token IDs are discrete integer identifiers from the tokenizer’s vocabulary. An ID is an index, not a meaning-rich vector.
- Embeddings are learned numerical vectors associated with tokens or other units.
- Contextual representations are vectors produced after model layers combine each token with information from its surrounding context.
That last distinction matters. The word “bank” begins with an input representation, but transformer layers can produce different contextual representations in “river bank” and “bank account.” This is context-sensitive processing, not evidence that the system understands the concept as a person does.
For the mechanics of splitting text, continue to Tokenization Explained. For the vector representation step, see Word Embeddings Explained.
One Example from Input to Output
Suppose a company wants to route the message “I was charged twice” to its billing team. Several valid designs could solve the same task:
- A rule could match phrases such as “charged twice” or “duplicate charge.”
- A classical classifier could convert the message into numerical features and predict a billing category.
- A transformer classifier could tokenize the message, build contextual representations, and assign a routing label.
- A hybrid system could classify the request, retrieve the relevant billing policy, apply account-specific rules, and draft a response.
The best design depends on the required accuracy, available data, latency and cost limits, need for traceability, privacy constraints, and consequences of an error.
Different Approaches and When They Fit
| Approach | Useful when | Important tradeoff |
|---|---|---|
| Rules and patterns | Requirements are explicit and traceability matters | Coverage can become difficult as language variation grows |
| Retrieval | The answer or evidence should come from a known corpus | Quality depends on indexing, query processing, and ranking |
| Classical machine learning | A focused task has suitable labeled data or engineered features | Feature choices may limit how well nuance transfers |
| Neural or transformer models | Complex contextual representations or generation are important | They can require more compute and careful evaluation |
| Hybrid systems | A production workflow needs multiple strengths | More components create more integration and monitoring work |
This is not an “old versus modern” contest. A narrow, deterministic rule can be the safest choice for one requirement, while a transformer may be better for another.
Example: How a Generative Language Model Works
When a user submits a prompt to a modern generative system, the prompt is converted into model inputs by the associated tokenizer. The model processes the current context and produces probabilities for possible next tokens. A decoding procedure selects a token, adds it to the context, and repeats the process until the response ends.
The system does not simply choose a predetermined “most likely word,” and fluent output does not demonstrate human-like thought or understanding. It generates token by token from learned statistical structure, the supplied context, system design, and decoding choices.
Learn more about the architecture in Transformers in NLP.
Why Language Remains Difficult
System designers must account for ambiguity, missing context, domain-specific vocabulary, sarcasm, cultural references, spelling variation, and differences across languages. Performance can also change when real-world input differs from development data.
Speech products add another boundary: spoken audio may first pass through automatic speech recognition, then through text or language processing, and perhaps through text-to-speech. NLP can be one part of that product without every audio-processing stage being NLP.
Evaluation Is Part of the Lifecycle
A practical lifecycle is: define the task → choose an approach and data → develop → validate → test → deploy → monitor. Evaluation is not one universal score. Classification, extraction, retrieval, translation, and generation require different measures, and important systems may also need human review, robustness checks, latency and cost monitoring, and analysis across languages or user groups. This matches NIST’s emphasis on measurement and evaluation across both AI technologies and their use contexts.
When failures appear after deployment, teams can revise rules, retrieval data, preprocessing, training data, model choices, thresholds, or safeguards and evaluate the updated system again. For a deeper introduction to task-specific measures, read Model Evaluation Metrics Explained.
Frequently Asked Questions
Does every NLP system use tokenization?
No. Tokenization is important for many model-based NLP systems, especially transformers, but simple rules, exact matching, or some retrieval operations may not use a learned model tokenizer.
Does an NLP model train on every request?
Usually not. Training or adaptation happens during development. At runtime, an already-built system processes new input to produce an output.
Are token IDs the same as embeddings?
No. Token IDs are integer identifiers. Embeddings are learned vectors that a model uses as numerical representations.
Are all modern NLP systems transformers?
No. Transformers are widely used, but rules, retrieval, classical machine learning, other neural architectures, and hybrid systems remain useful.
Sources and Further Reading
- Hugging Face: Tokenizer documentation
- Vaswani et al.: Attention Is All You Need
- NIST: AI Measurement and Evaluation
Next Lesson: Tokenization Explained
You now have the system-level view: different NLP designs follow different paths, and model development is separate from runtime use. The next lesson explains how many language models divide text into processable units.
Need the broader overview first? Return to What Is Natural Language Processing (NLP)?
Last reviewed: September 2026.