A neural network is a machine-learning model built from layers of connected computational units. These units transform numerical inputs using learned weights, biases, and nonlinear functions so the network can approximate useful relationships in data. Google’s current Machine Learning Glossary provides a useful technical reference for neural-network, neuron, activation-function, and backpropagation terminology. Neural networks power many modern AI systems, including image recognition, speech processing, language models, recommendation systems, and generative AI. Despite the biological inspiration behind their name, artificial neural networks are mathematical models, not digital brains.
Lesson 8 · Published March 12, 2026 · Updated September 1, 2026
Previous Lesson: What Is Deep Learning
What Is a Neural Network?
A neural network takes an input, passes it through one or more layers of computation, and produces an output. A simple network may contain:
- An input layer that receives features.
- One or more hidden layers that transform those features.
- An output layer that produces a prediction or representation.
Each connection has a learnable weight. Many units also include a learnable bias. During training, an optimization process adjusts these parameters to reduce a loss function.
How an Artificial Neuron Works
A simplified artificial neuron performs several steps:
- Receive numerical inputs.
- Multiply each input by a weight.
- Add the weighted inputs and a bias.
- Apply an activation function.
- Pass the result to later units or the output.
A common mathematical form is: output = activation(weighted sum of inputs + bias) The weights determine how strongly different inputs influence the unit. The activation function introduces nonlinearity, allowing a network to model relationships that a purely linear stack could not represent. The term neuron is an analogy. Artificial neurons are much simpler than biological neurons and should not be treated as realistic simulations of the brain.
Layers in a Neural Network
Input layer
The input layer represents the information provided to the model. Examples include:
- Pixel values or image patches.
- Numerical business features.
- Encoded words or tokens.
- Sensor readings.
- Audio features.
Hidden layers
Hidden layers perform intermediate transformations. Earlier layers may learn relatively simple patterns, while later layers can represent more task-relevant combinations. This “simple to complex” description is a useful intuition, but the exact representations learned by modern networks depend on architecture, data, objective, and training process.
Output layer
The output layer depends on the task. It might produce:
- Class probabilities.
- A numerical prediction.
- Token probabilities for language generation.
- An embedding vector.
- Control values.
What Are Weights and Biases?
Weights are learned parameters that control how inputs influence later computations. Biases allow units to shift their activation independently of the input values. A network can contain thousands, millions, billions, or more parameters depending on its size and architecture. Parameters are not individually programmed with human-readable rules. They are adjusted through optimization based on the training objective.
What Are Activation Functions?
Activation functions introduce nonlinear behavior into neural networks. Without nonlinear activations, stacking many ordinary linear layers would still collapse into a linear transformation and could not represent the complex relationships associated with deep networks. Common activation functions include:
ReLU
The rectified linear unit, or ReLU, outputs zero for negative inputs and passes positive inputs through. Variants of ReLU are widely used in neural networks.
Sigmoid
The sigmoid function maps values into a range between 0 and 1. It is often used for certain binary outputs, although it is less common than ReLU-like functions inside many deep hidden layers.
Tanh
Tanh maps values between -1 and 1 and appears in some recurrent and older network architectures.
GELU and related activations
Modern transformer architectures commonly use functions such as GELU or related variants.
How Neural Networks Learn
Neural-network training usually involves four broad steps.
1. Forward pass
Input data moves through the network to produce a prediction.
2. Calculate loss
A loss function measures how well the model’s output matches the training objective. Examples include cross-entropy for many classification tasks and mean squared error for some regression tasks.
3. Backpropagation
Backpropagation efficiently calculates how changes in the network’s parameters would affect the loss by applying the chain rule. Backpropagation does not decide how parameters should change by itself. It computes gradients: signals that show how the loss would change if a parameter changed. An optimizer then uses those gradients to update the parameters. The official PyTorch optimization tutorial demonstrates this separation between gradient computation and the optimizer step.
4. Optimization
An optimizer such as stochastic gradient descent or Adam uses those gradients to update the parameters. This process repeats over many batches of data. The combination of a loss function, backpropagation, and an optimizer allows the network to gradually fit useful patterns for the training objective.
Next Lesson: Continue to How Deep Learning Works for the complete training process, including batches, epochs, validation, and regularization.
Training vs Inference
Training and inference are different phases.
Training
During training, the model’s parameters are adjusted based on data and an objective. Training may require substantial compute, memory, and data preparation.
Inference
During inference, the trained parameters are usually held fixed while the network processes new inputs and produces outputs. Inference can still involve complex surrounding systems—retrieval, tools, decoding, filtering, or business rules—but ordinary inference does not mean the model is continuously retraining itself from every interaction.
Neural Networks vs Deep Learning
A neural network does not have to be “deep.” Deep learning refers to machine-learning approaches built around multilayer or deep neural networks and learned representations.
| Term | Meaning |
|---|---|
| Neural network | A model built from connected computational units and learnable parameters. |
| Deep neural network | A neural network with multiple learned layers. |
| Deep learning | The broader machine-learning approach centered on deep neural networks and learned representations. |
Read What Is Deep Learning?.
Major Neural Network Architectures
Neural networks come in many forms. There is no single architecture used for every AI problem.
Feed-forward networks and multilayer perceptrons
Information moves from input toward output through a stack of layers. These networks are useful for many structured-data and general approximation tasks. NIST defines feedforward neural networks as artificial neural networks whose node connections proceed from one layer to the next without forming cycles.
Convolutional neural networks
Convolutional neural networks, or CNNs, use operations designed to exploit local spatial structure. They became especially important in computer vision and remain useful in many vision systems. Read Computer Vision Explained.
Recurrent neural networks
Recurrent neural networks, including LSTMs and GRUs, were designed for sequential data and maintain information across steps. They remain important historically and in some applications, although transformers have replaced them in many large-scale language and sequence-modeling systems.
Transformers
Transformers use attention mechanisms to model relationships among elements in a sequence or set of representations. They are foundational to modern large language models and are also widely used in vision, audio, multimodal AI, and other domains. See Transformers Explained.
Autoencoders
Autoencoders learn to encode data into a representation and reconstruct it. Variants are used for representation learning, compression, anomaly detection, and generative modeling.
Graph neural networks
Graph neural networks operate on graph-structured data such as molecules, social networks, knowledge graphs, or transportation networks.
How Neural Networks Are Trained Today
Not all neural networks are trained from scratch with manually labeled datasets. For a beginner, modern approaches are easiest to understand in three groups:
- Learning objectives: supervised, self-supervised, or reinforcement-based learning.
- Adaptation: a pretrained model can be transferred or fine-tuned for a new task.
- Advanced methods: modern systems may combine additional training techniques when the application requires them.
A large model may be pretrained on broad data and later adapted to a narrower application. Read Supervised Learning Explained and Types of Machine Learning.
Where Neural Networks Are Used
Computer vision
Neural networks can classify images, detect objects, segment scenes, estimate depth, and learn visual representations.
Natural language processing
Neural networks power translation, search, classification, summarization, question answering, and language generation. See What Is Natural Language Processing?.
Speech and audio
Networks are used for speech recognition, speech synthesis, speaker identification, audio classification, and music-generation systems.
Generative AI
Generative models use neural networks to produce text, images, audio, video, code, and other outputs. Read What Is Generative AI?.
Forecasting and structured data
Neural networks can also be used for tabular and time-series problems, although simpler models such as gradient-boosted trees or linear methods may outperform them depending on the dataset and constraints.
Advantages of Neural Networks
Flexible function approximation
Neural networks can model complex nonlinear relationships.
Representation learning
Deep networks can learn useful internal representations from model inputs, reducing some forms of manual feature engineering.
Scale
Many neural architectures benefit from larger datasets, models, and compute when scaling is done carefully.
Transferability
Pretrained networks can often be adapted to new tasks, reducing the amount of task-specific training required.
Limitations of Neural Networks
Compute and energy cost
Large neural networks can be expensive to train and serve.
Data requirements
Training from scratch can require substantial data, although pretraining and transfer learning can reduce the amount of labeled task-specific data needed.
Interpretability
Large networks can be difficult to explain in simple human terms. Interpretability tools can provide useful evidence, but they do not automatically make every decision transparent.
Overfitting
Neural networks can memorize training patterns that do not generalize. See Overfitting vs Underfitting.
Distribution shift
Performance may decline when deployment data differs from training conditions.
Bias and data quality
Networks can reflect biases, errors, omissions, and imbalances in training data and objectives.
Security and robustness
Neural systems can be vulnerable to adversarial inputs, data poisoning, model extraction, robustness failures, and other security threats.
Do Neural Networks Think Like the Human Brain?
No. The terminology was inspired partly by neuroscience, but modern artificial neural networks are engineering and mathematical systems. They do not reproduce the structure, chemistry, learning process, or cognitive capabilities of a biological brain. Calling a computational unit a “neuron” is a convenient analogy, not evidence that the system thinks or experiences the world like a person.
Frequently Asked Questions
What is a neural network in simple terms?
It is a machine-learning model made of layers of connected mathematical operations whose parameters are adjusted during training to perform a task.
What does backpropagation do?
Backpropagation calculates gradients showing how model parameters contribute to the loss. An optimizer then uses those gradients to update the parameters.
Are all neural networks deep learning?
No. Neural networks can be shallow. Deep learning refers to approaches using neural networks with multiple learned layers and complex representation learning.
Are transformers neural networks?
Yes. Transformers are a neural-network architecture built around attention mechanisms.
Do neural networks learn during every prediction?
Usually not. Standard inference uses parameters learned during training. Some systems can include online learning or memory components, but that is an additional design choice.
Do neural networks require huge labeled datasets?
Not always. Modern systems can use self-supervised pretraining, synthetic data, transfer learning, weak supervision, and other methods that reduce reliance on large manually labeled datasets.
Continue the Learning Path
Next Lesson: How Deep Learning Works
Now that you understand neural-network components and the four-step training overview, Lesson 9 explains the complete deep-learning training process.