Saddle Points in Optimization: What They Are and Why They Matter

Even if every partial derivative in your function is zero, you're not guaranteed to be near a minimum.

Optimization is the search for a function's lowest or highest value, and every method you use looks for a spot where the gradient is flat. But a flat gradient doesn't guarantee you've found the best answer. Sometimes it means the function goes up in one direction and down in another, right at the same point.

That point is called a saddle point and it goes down one way, up the other, and is flat all around it. Recognizing saddle points matters because your algorithm can get stuck there and mistake a flat gradient for an actual minimum.

In this article, I'll walk you through how to spot a saddle point mathematically and why it often confuses optimization algorithms.

What Is a Saddle Point?

At a saddle point, the gradient is zero, just like at a minimum or a maximum. But a saddle point is neither one.

The function curves upward in some directions and downward in others, right at the same point. If you go along one axis, the value goes up, and if you go along a different axis, it drops. That's the idea behind the name - it borrows from the shape of a horse saddle, which curves up toward the front and back and down toward the sides.

Here's a simple two-variable function I'll use to show what this looks like:

Two-variable function

At the origin, (0, 0), both partial derivatives equal zero, so it's a critical point (also called a stationary point). Move along the x-axis, and f increases because of the x^2 term. Move along the y-axis, and f decreases because of the -y^2 term.

If you plot this function in 3D, the surface curves up in one direction and down in the other, forming the shape a saddle point is named after.

3D surface plot of f(x, y) = x^2 - y^2

A contour plot of the same function tells the same story from above. The contour lines around the origin look like hyperbolas, not the closed loops you'd see around a minimum or maximum.

Contour plot of f(x, y) = x^2 - y^2

The exact definition changes a bit depending on the field. In multivariable calculus, a saddle point is a critical point that isn't a local minimum or maximum. In optimization and game theory, the term can mean something different, tied to minimax problems instead of curvature. I'll cover that distinction later in this article.

Saddle Points vs. Minima and Maxima

Before you get into how to identify saddle points mathematically, it helps to see how they compare against the other two kinds of critical points.

Local minimum

At a local minimum, the function increases in every nearby direction. Imagine a bowl - when you step away from the bottom in any direction, you go up. The function f(x, y) = x2 + y2 behaves this way.

Local maximum

At a local maximum, the function decreases in every nearby direction. It's the mirror image of a minimum, a dome instead of a bowl. The function f(x, y) = -x2 - y2 decreases along every direction as you move away from the origin.

Saddle point

At a saddle point, the function increases in some directions and decreases in others. That's what makes f(x, y) = x2 - y2, the example from the previous section, a saddle point instead of a minimum or a maximum: it goes up along the x-axis and down along the y-axis, at the same point.

Here's how the three compare side by side:

Side-by-side 3D surface plots of a local minimum, local maximum, and saddle point

How to Find a Saddle Point

Finding a saddle point comes down to taking the derivatives, finding where they vanish, and then looking at how the function curves around that point.

I'll use the same function from before:

Reference function

Here's the process, step by step.

1. Compute the first partial derivatives

Take the derivative of f with respect to x, treating y as a constant, then do the same for y:

Computing the first partial derivatives

2. Set them equal to zero

Solve for the point where both partial derivatives vanish at once:

Setting derivatives to zero

That gives x = 0 and y = 0, so (0, 0) is a critical point. First derivatives won't tell you what kind.

3. Compute the second derivatives

You need three of them: the derivative of f_x with respect to x, the derivative of f_y with respect to y, and the mixed derivative:

Second derivatives

4. Classify the critical point using curvature 

f_xx = 2 is positive, so f curves upward along the x-axis. f_yy = -2 is negative, so f curves downward along the y-axis. One direction curves up, the other curves down, at the same point - that's the saddle point.

This four-step process works for any twice-differentiable function.

In the following two sections, I'll formalize step 4 with tools that scale past two variables: the Hessian matrix and the second derivative test.

How the Hessian Matrix Identifies Saddle Points

The Hessian matrix includes all the second derivatives from the last section into a single object, and it's the tool that tells you whether a critical point is a minimum, a maximum, or a saddle.

For a two-variable function, the Hessian looks like this:

Hessian for a two-variable function

Each entry describes curvature: f_xx and f_yy describe curvature along the x and y axes, and f_xy describes how those two directions interact. If you plug in the second derivatives from the previous section, the Hessian for our example function looks like this:

Hessian for the reference function

A matrix on its own doesn't tell you much. What matters are its eigenvalues - the numbers that describe how the function curves along a specific set of directions called eigenvectors.

• A positive eigenvalue means the function curves upward along its eigenvector's direction
• A negative eigenvalue means the function curves downward along that direction
• A mix of positive and negative eigenvalues means the function curves up in some directions and down in others (a saddle point)

Because our example's Hessian is diagonal, the eigenvalues are right on the diagonal: 2 and -2. Their eigenvectors point along the x-axis and y-axis, which is exactly the curvature you found by hand in the last section. f curves up along x because the eigenvalue there is positive, and down along y because the eigenvalue there is negative.

Most functions won't have a diagonal Hessian, so the eigenvectors won't line up neatly with the x and y axes, and you'd need to solve for them directly instead of reading them off the matrix. But one positive eigenvalue and one negative eigenvalue means you're standing on a saddle point.

The Second Derivative Test for Saddle Points

You don't need to compute eigenvalues by hand every time. For two-variable functions, there's a shortcut - the second derivative test - built from the Hessian's determinant.

Here's the determinant, using the same second derivatives from before:

The determinant

D is the determinant of the Hessian, and its sign - combined with the sign of f_xx - classifies the critical point:

• D > 0 and f_xx > 0: Local minimum

• D > 0 and f_xx < 0: Local maximum

• D < 0: Saddle point

• D = 0: Inconclusive

For our example, f_xx = 2, f_yy = -2, and f_xy = 0:

Second derivative test for the reference function

D is negative, so the test confirms that (0, 0) is a saddle point.

This test is the eigenvalue rule from the last section, rewritten in a form you can compute without solving for eigenvectors. D is the product of the two eigenvalues, and f_xx + f_yy is their sum. When D is negative, the eigenvalues have to carry opposite signs, and that's what a saddle point is. When D is zero, at least one eigenvalue is zero, so the curvature in that direction is flat, and neither the test nor the eigenvalues alone can tell you what's going on there.

Saddle Points in Optimization

Everything up to this point has been about identifying a saddle point once you already have a function in hand. But when it comes to optimization, you're searching for the best point without knowing where it is in advance, and gradient-based methods use zero gradient to decide when to stop.

That's exactly where saddle points cause trouble.

A gradient-based optimizer moves in the direction that reduces the objective function fastest, and it keeps going until the gradient gets close to zero. But a near-zero gradient doesn't tell the algorithm whether it found a minimum or a saddle point.

Directions of positive curvature push the function up, and the algorithm has already learned that stepping there makes things worse. The directions of negative curvature are still there, and they're the way out - but an algorithm that only trusts the gradient can miss them, because right at a saddle point, the gradient in those directions is close to zero too.

This is also where optimization slows down.

Gradient magnitude tends to shrink as you approach any stationary point, so an algorithm that scales its steps to the gradient size takes small steps near a saddle, whether or not it's near the minimum it's actually looking for.

Finding a stationary point isn't the same as finishing an optimization. You still need a way to tell that a near-zero gradient belongs to a saddle point and not a minimum, and a way to find the direction that keeps improving the objective once you know it.

Saddle Points in Machine Learning

A neural network's loss function is a function of every weight and bias in the network, sometimes millions or billions of dimensions. It's far away from a two-variable toy example.

In an analysis of high-dimensional non-convex optimization, Dauphin and colleagues argue that as the number of dimensions grows, critical points are far more likely to be saddle points than local minima or maxima. Each dimension adds another direction the curvature can go either way, and it only takes one direction of negative curvature to turn a critical point into a saddle instead of a minimum.

That matters during training because gradient descent can only see the gradient it's currently standing on.

Near a saddle point, that gradient gets small in every direction - the same signal you'd see approaching an actual minimum. Training can slow down or get stuck in a region that isn't optimal at all, and there's no way to tell from the loss curve alone whether you're near a minimum or stuck on a saddle.

This is why saddle points come up so often alongside local minima when people talk about neural network training. But they aren't the only obstacle. Vanishing gradients, ill-conditioned curvature, and the scale of the search space all make high-dimensional optimization hard.

How Optimization Algorithms Handle Saddle Points

None of this means gradient-based training is doomed near a saddle point. There are a few techniques that can help you push through them, each from a different angle.

Stochastic gradient descent

Stochastic gradient descent computes the gradient from a random subset of the training data instead of the whole dataset, and that subset changes on every step.

The noise this introduces means the estimated gradient is rarely exactly at zero, even in a region where the true gradient is close to zero. That noise can be enough to push the optimizer off a saddle point and into a direction of negative curvature, instead of stalling exactly where the gradient vanishes.

Momentum

Momentum keeps a running average of past gradients and uses that, instead of just the current one, to decide the next step.

Around a saddle point, this matters because the optimizer arrives already moving in some direction, carried by the gradients from before it got there. That accumulated motion can carry it through the flat region around the saddle, rather than slowing down the moment the local gradient goes near zero.

Second-order methods

Second-order methods use curvature information directly.

By working with the Hessian or an approximation of it, these methods can tell the difference between directions of positive curvature and directions of negative curvature at a critical point. That's the distinction a saddle point hinges on, so a second-order method can identify the descending direction and step into it, instead of treating every near-zero gradient the same way.

Saddle Points vs. Related Concepts

A few terms get mixed up with saddle points often enough that they're worth addressing.

Saddle point vs. local minimum

A local minimum curves upward in every direction near it, so it's a consistent positive curvature, all the way around. A saddle point only curves upward in some directions and curves downward in others. The moment you find even one direction with negative curvature, you're not looking at a minimum anymore.

Saddle point vs. local maximum

The same logic runs the other way for a local maximum. Curvature stays negative in every direction near it. A saddle point adds at least one direction of positive curvature alongside the negative ones.

Saddle point vs. inflection point

An inflection point is a one-dimensional idea. It's where a curve switches from concave up to concave down, or the other way around, along a single variable.

A saddle point needs at least two variables, because the whole concept depends on comparing curvature across different directions at the same point. A function of one variable can have an inflection point, but it can't have a saddle point, as there's no second direction to curve the other way.

Saddle point vs. stationary point

Every saddle point is a stationary point, since the gradient is zero there. But the reverse isn't true. A stationary point is just any point where the gradient is zero - it could be a local minimum, a local maximum, or a saddle point. You need curvature information from the Hessian or the second derivative test to know which one you've got.

Here's a more visual overview of how they compare:

Saddle points compared with other concepts

Saddle Points in Minimax Optimization

Everything so far has used "saddle point" to describe a critical point's curvature. There's a second, older meaning that comes from game theory, and it shows up in modern machine learning too.

In minimax optimization, one player minimizes a function while another maximizes it, and a saddle point is the pair of values where neither player can improve their outcome by changing their own move alone.

Game theory uses this to describe equilibrium strategies in competitive games. Adversarial optimization borrows the same structure for problems where one process searches for the worst case that another process has to defend against.

Generative adversarial networks (GANs) are the best machine learning example. A generator minimizes its loss by producing convincing fake data, while a discriminator maximizes its ability to tell real data from fake. Training a GAN means searching for a saddle point in this minimax sense, or a balance where the generator and discriminator have each found their best response to the other.

This minimax saddle point is related to the critical-point saddle point from earlier in this article, but the two aren't interchangeable.

The critical-point version describes curvature at a single point of one function. The minimax version describes an equilibrium between two competing objectives. Both use the same word because both involve a mix of "up" and "down" behavior, but conflating them won't get you far if you try to apply a Hessian-based test to a two-player game.

Common Mistakes When Working with Saddle Points

Most saddle point confusion comes down to a couple of repeated mistakes. Here's what you need to know.

Assuming a zero gradient means a minimum

A zero gradient only tells you that you've found a critical point, not what kind. Check the curvature before you assume you're done since a saddle point has a zero gradient too.

Classifying a point using only first derivatives

First derivatives find critical points but they can't classify them. You need the second derivatives, through the Hessian or the second derivative test, to tell a minimum from a maximum from a saddle point.

Ignoring different directions of curvature

A function can curve upward along the x-axis and still be a saddle point overall, if some other direction curves downward. Check every direction, or use the eigenvalues of the Hessian, which cover all of them at once.

Confusing saddle points with inflection points

An inflection point describes a curvature change in a single variable. A saddle point describes mixed curvature across multiple variables at the same point. They're related in the general idea, but they don't apply to the same kind of function, and the math behind them isn't interchangeable.

Treating all uses of "saddle point" as identical

The critical-point saddle point and the minimax saddle point share a name, not a definition. Before you apply a Hessian-based test or an eigenvalue argument, check which meaning is actually in play - game theory and adversarial optimization use the minimax version, and it doesn't reduce to a curvature check on a single function.

Conclusion

A saddle point curves upward in some directions and downward in others, at the same coordinates - that's why it's neither a minimum nor a maximum, even though the gradient goes to zero there just like it does at both.

Gradients find the critical point. The Hessian's eigenvalues, or the second derivative test, tell you which kind you've got. In high dimensions, that distinction matters a lot because saddle points outnumber local minima, and training algorithms like SGD and momentum all exist partly to get past them.

One last thing to keep separate is that the minimax saddle point from game theory and GANs shares the name but not the math - so make sure you know which one you're referring to.

If you find the idea behind saddle points interesting, you'll love our Deep Learning Track in Python. Along the theory, you'll use PyTorch to create neural networks for different kinds of problems.