What is the gradient descent method?
Gradient descent is one of the fundamental techniques used in machine learning and artificial intelligence. It allows models to gradually adjust their parameters so that their predictions become increasingly accurate.
Model training can be described quite simply, even though complex mathematics is at work behind the scenes. We provide a question with the correct answer, the model responds based on randomly initialized parameters, and then we compare the result with the expected value. The difference between them shows how much the model was mistaken. Based on this, we adjust the parameters so that the next attempt is at least slightly better. The entire process is repeated many times until the model begins to recognize patterns.
This is how gradient descent works: it indicates the direction in which the parameters should change and the magnitude of that change, which is compared to descending toward the lowest point of a valley. Since the problems are too complex to calculate the parameters exactly, the method uses approximations and gradually moves closer to the best solution.
This is how most modern models work, from neural networks to regression techniques, which analyze text, numbers, and images. Without gradient descent, the development of today’s AI would be practically impossible.
Table of Contents
Gradient descent, or descending along the slope.
The easiest way to put it is to use an analogy. Suppose you are standing at the top of a mountain in the dark. You cannot see your surroundings, but you can sense which direction the terrain slopes downward. Each step down brings you closer to the valley, where the model’s error is smallest. Gradient descent works in a similar way: it uses local information about the slope of the surface to determine the direction of movement.
This intuition allows the method to work even when the cost function is complex and multidimensional. The model does not need to know the entire mathematical landscape; information about the local slope is sufficient.
The analogy of descending a slope is very interesting, but how can it be translated into numbers that then generate text and images or perform analyses?
Training AI models can be reduced to a simple situation: we have a formula, for example^=ax+b,\hat{y} = a x + b,
and we want it to produce, for a given xxx, a result as close as possible to the actual yy. The problem is that we do not know good values for aa and bb. Initially, we set them randomly, for example, a=0a = 0 and b=0b = 0. If we substitute x=2x = 2x=2, the model will predict y^=0⋅2+0=0,\hat{y} = 0 \cdot 2 + 0 = 0,
but let’s assume that the correct answer is y=5y = 5. The difference between 0 and 5 is the error, which tells us how much the model is mistaken.
Based on this difference, we decide how to modify the parameters. If the result was too small, aa and bb should be increased. If it was too large, they need to be decreased. We do not calculate the ideal values because the problem is too complex, even in larger models with millions of parameters. Instead, we make approximations: we adjust the parameters by a small fraction of the error and check the effect. We repeat this many times until the model starts getting closer and closer to the truth.
This is where the analogy of descending a slope fits perfectly. The direction of descent tells us which parameters need to be changed: whether aaa should increase or decrease; whether bbb should be raised or lowered. The “steepness” of the slope, in turn, tells us how much we should change them: a small error impact means a gentle adjustment, while a large impact requires a more substantial change.
The model cannot see the entire terrain or know the ideal values, but thanks to these local cues, the direction and magnitude of the changes, it can gradually approach a solution in which the error is as small as possible.
Graphic: Tomasz59 Gradient method – trajectories from the starting point to the minimum determined using a constant step (step size = 0.05, 381 steps – red trajectory) and an optimally selected step in the steepest descent method (117 steps – blue trajectory). It can be seen that with the optimal step selection method, the point quickly reaches the vicinity of the minimum (then slows down because the minimum is very flat).
What Happens During One Gradient Descent Step
The learning process begins with the cost function. This function determines how large the model’s error is. For example, in linear regression, the mean squared error is often used, that is, the average difference between predictions and actual values.
Gradient as the Direction of Movement
Gradient is a set of derivatives that indicates the direction of the fastest increase in the cost function. Since we want to reduce the error, we move in the opposite direction. The gradient determines the direction in which the model’s weights should be adjusted so that the cost function is smaller.
It is worth knowing that the gradient can vary in magnitude. A large gradient indicates a steep slope and leads to larger adjustments. A small gradient signals a flat surface and causes only slight changes in the parameters.
Model Weight Update
The basic formula for updating weights is as follows: new weight = old weight minus the learning rate multiplied by the gradient.
It is a simple principle, but repeating it over thousands or millions of iterations gradually reduces the error and brings the model closer to the optimal parameters.
After each change, a new error is calculated and another step is performed. The process continues until the model stabilizes or meets specified stopping criteria, such as a minimal change in the error.
Gradient Descent and backpropagation
Gradient descent and backpropagation are two components of the same process. The former determines which direction to move the parameters, while the latter determines how to calculate this direction for each parameter throughout the entire network. Together, they form a mechanism that makes it possible to train models with many layers without manually calculating thousands of derivatives.
This can be described in a few steps. First, gradient descent needs information about how the error changes with the parameters. The method itself only says that we must move in the direction opposite to the gradient in order to reduce the error. It does not, however, say how to calculate the gradient of a network with hundreds of layers, nonlinear activation functions, and millions of weights. This is where backpropagation comes in: it calculates the necessary derivatives in an orderly manner (from the network’s output toward its input), using the chain rule and storing intermediate results so that nothing has to be calculated twice.
In practice, it works like this: after performing one forward pass through the network, you obtain an output and an error. Backpropagation then breaks this error down into smaller components, assigning a portion of it to each weight and neuron. In the end, you have complete information about how each individual parameter change would affect the error. Gradient descent uses this data to update the weights by a small amount determined by the learning rate.
In this way, backpropagation provides the gradient, while gradient descent performs an update step. The two elements therefore work together, forming a closed loop: the forward pass through the network, error calculation, backpropagation, parameter update, and another forward pass. Iterations of this cycle make it possible to approach a solution that cannot be expressed as a single equation.
Gradient Descent Using a Simple Numerical Example.
Let’s return to our simple example, but this time we’ll examine it more closely. y^=wx+b\hat{y} = w x + b
and we want this formula to produce a result as close as possible to the true value yy. The problem is that we do not know the correct values of the variables ww and bb. We only have the input data xx and the known outputs yy. Your task is to teach the formula the appropriate parameter values.
Gradient descent does this in three simple steps, which it repeats many times.
Step 1: we calculate the result of the formula
First, we substitute the current values of the variables. At the beginning, they are usually random or zero.
If w=0w = 0 and b=0b = 0, then the model’s prediction for any xx is y^=0⋅x+0=0\hat{y} = 0 \cdot x + 0 = 0
Step 2: calculate the error
Since we know the true result yy, we can calculate how far off we were. Example: the true result is y=5y = 5y=5, the model predicted 00. The error is the difference
y^−y=0−5=−5\hat{y} – y = 0 – 5 = -5
You can treat this as feedback: the model says what came out of its formula, and we compare it with the truth.
Step 3: we improve the variables based on the error
This is where the essence of gradient descent begins. If the model predicts too little, we update the variables so that the predictions are higher in the next step. If it predicts too much, we change the variables in the opposite direction.
In practice, we apply a simple rule:
new value = old value − α × error impact(\text{new value} = \text{old value} – \alpha \times \text{error impact})
where α\alpha is the learning rate, that is, how large a step we want to take. The learning rate is simply a number that is generally chosen manually. A learning rate that is too small makes training very slow, but progress is noticeable with each iteration; a learning rate that is too large may cause the model to “jump” around the result and never “come down.”
There are also techniques that automatically adjust the learning rate to specific needs, known as adaptive optimizers, such as Adam, RMSProp, and Adagrad. They modify the effective learning rate during training, adapting it to the behavior of the gradients. However, even with these methods, we still need to choose an initial value.
What figures are we talking about?
0.1 – a high learning rate, generally used in simple models or during the early stages of training;
0.01 – a very popular value in classical methods and many optimizers;
0.001 – the most common choice in modern models, especially neural networks;
0.0001 and lower – used when the model is highly sensitive or the gradients are unstable.
What I call the “impact of the error” here is precisely the gradient. The gradient is simply information: how much a given variable contributed to the error. If the impact is large, we change it more. If it is small, the change is subtle.
Graphic: Tomasz59 Gradient descent method – a step size that is too large, =0.15, leads to oscillations near the minimum without reaching it (red trajectory). For comparison, the trajectory with an optimally chosen step size is shown (blue trajectory).
Why it works
Each time, we adjust the variables based on what the model got wrong. If the error is large, the variables are adjusted more strongly; as it decreases, the changes become smaller. After many repetitions, the parameter values begin to “fit” the data, and the predictions become increasingly accurate.
This can be compared not to descending a slope, but to guessing:
- We guess the values of the variables.
- We check how far off we were.
- Based on the error, we adjust the guessed values.
- We repeat until the guesses become very accurate.
Gradient descent is simply the systematic improvement of guessed variable values until the formula stops making mistakes.
Common Problems When Working with Gradient Descent
Although the method is universal and simple, difficulties arise in practice.
Local Minima and Flat Surfaces
The cost function for real-world models is not a smooth bowl, but something resembling a complex landscape. Sometimes a model gets stuck in a place that is good, but not the best. Local minima can be avoided by using random data shuffling, appropriate learning variants, or additional mechanisms.
Flat regions cause the gradient to be very small, and the model takes minimal steps. This is a natural phenomenon in large neural networks, which can have areas where the derivatives are close to zero.
Instability and Oscillations
If the learning rate is too high, the model may jump back and forth across the minimum, resembling bouncing off a wall. In such situations, smoothing mechanisms such as momentum are used. They reduce the risk of abrupt changes in direction.
Does Gradient Descent Remain the Foundation of Artificial Intelligence
Despite the development of increasingly advanced optimization techniques, gradient descent remains the foundation of model training. Its strength lies in its simplicity, flexibility, and ability to operate in spaces with a vast number of parameters. Regardless of how complex algorithms become, the essence of the process remains the same: to reduce the error step by step until the model finds the most accurate solution.
Gradient descent is an example of how a simple idea can become one of the pillars of modern technology. Thanks to it, models learn to see, listen, and understand. Each step down the slope of the cost function brings us closer to intelligent tools that make everyday life easier and expand the boundaries of AI’s capabilities.