Regularization in Machine Learning: L1, L2, and Elastic Net Explained

So, you’ve trained a model that gets every training example nearly perfect, but fails on new data? We’ve all been there.

That's a high-level definition of overfitting. Your model didn't learn the actual pattern, but instead, it memorized the training data. In a production environment with new and unseen data, the model would make predictions you wouldn’t trust. The more that real-world data drifts from training samples, the worse this gets.

Regularization fixes this by adding a penalty to the loss function. That penalty discourages complex models. In other words, it's the mechanism that keeps your model from fitting every data point and forces it to generalize instead.

In this article, I'll walk you through the intuition behind regularization, the most common methods - L1, L2, and Elastic Net - and how to pick the right one for your use case.

If you want to understand why and how machine learning models fail in production, read our Bias-Variance Tradeoff blog post.

What is Regularization in Machine Learning

Regularization is a technique that adds a penalty term to your model's loss function to discourage complexity.

Without this penalty term, a model is flexible enough to fit the training data as closely as it wants. That includes the noise and outliers. Regularization adds a cost on that flexibility. The more complex the model wants to be, the higher the penalty it gets.

Your model's loss function normally measures the difference between predicted and actual values. Regularization adds an extra term to that equation, one that grows as the model's coefficients grow. The model now has to balance two competing objectives: fit the training data, and keep the coefficients small.

That balance is what controls model flexibility.

A highly flexible model can twist itself into any shape to fit training data. Regularization smoothes it back to a simpler shape - one that is more likely to hold up on data the model hasn't seen before.

Why Regularization Is Needed

Every model you train sits somewhere between two unusable models: one that’s too simple and one that’s too complex.

A model that's too simple doesn't “get” the real patterns in your data. It misses the signal. That's underfitting - the model performs poorly on training data and on new data.

A model that's too complex does the opposite. It fits every detail in your training data, including the noise. That's overfitting - the model performs great on training data, but fails on new data because it memorized the wrong things.

Take polynomial regression as a concrete example. A degree-3 polynomial fit through data that shows a gentle curve means you’ll likely fit to a correct pattern. But a degree-15 polynomial through the same data leads overfit - the curve goes through every data point, but makes random predictions in between.

The chart below shows what that looks like in practice.

Just-right versus too complex model

This is the bias-variance tradeoff.

Simple models have high bias - they make strong assumptions that miss real patterns. Complex models have high variance - they're too sensitive to the specific training samples they saw, and small changes in the data produce very different models.

Regularization is what helps you get the best of both. It doesn't eliminate complexity, but it penalizes it. As a result, your model learns has a better chance to learn the real signal.

How Regularization Works

Every model learns by minimizing a loss function - a measure of how wrong its predictions are. Without regularization, the model's only job is to minimize that error. It'll do whatever it takes, including growing large coefficients that fit the training data but don’t generalize.

Regularization changes the objective. Instead of minimizing error alone, the model now minimizes this:

How regularization works

The penalty term is a function of the model's coefficients. Large coefficients bring the penalty up. To keep the total cost low, the model is forced to keep its coefficients small - which means simpler, more generalizable solutions.

The λ (lambda) controls how much the penalty matters. A higher λ adds more pressure on the model to stay simple. A lower λ lets the model focus more on fitting the data. You'll see how to tune this in the Choosing the Regularization Strength section below.

Types of Regularization In Machine Learning

There are a couple of ways to penalize model complexity. Each one puts pressure on the coefficients in a different way, which means they're suited to different situations.

L2 regularization (Ridge regression)

L2 regularization penalizes the squared value of each coefficient. The larger a coefficient, the more it contributes to the penalty - and the harder the model works to shrink it.

L2 regularization

The key word here is shrink. L2 pushes all coefficients toward zero, but never quite reaches zero. Every feature stays in the model, just with a smaller weight. That makes Ridge a good default when you believe most of your features are relevant and you want a stable, well-behaved model.

L1 regularization (Lasso regression)

L1 regularization penalizes the absolute value of each coefficient instead of the square.

L1 regularization

That small difference has a big consequence. L1 can push coefficients all the way to exactly zero, which means it removes features from the model. You can think of this as automatic feature selection. In other words, Lasso regularization can simplify your model by removing features.

L1 vs. L2 regularization

The core difference comes down to sparsity. L1 produces sparse models - only a subset of features get through. L2 produces dense models - all features remain, with smaller weights.

That affects interpretability too. A Lasso model with 5 active features is easier to explain than a Ridge model with 50 features all contributing a little. But Ridge tends to be more stable when features are correlated with each other, since it spreads the weight across them rather than arbitrarily picking one.

Here's a quick overview of the differences:

L1 versus L2 regularization

If you want to see how these compare in Python, our Lasso and Ridge Regression in Python tutorial has you covered.

Elastic Net regularization

Elastic Net combines L1 and L2 into a single penalty term.

Elastic Net regularization

The idea is to get the best of both: feature selection of L1 and stability of L2. This is handy when you have correlated features and still want some of them dropped. Lasso alone tends to pick one feature from a correlated group and ignore the rest. Elastic Net is more likely to keep a few of them while still removing irrelevant ones.

Regularization in Different Models

Regularization shows up across many machine learning models, but in different forms. Let me show you what these are.

Linear regression is where most people first see regularization. When you add L2 regularization to linear regression and you get Ridge regularization. Likewise, adding L1 gets  you Lasso regression. The math is the same as described above - a penalty term added to the least squares loss.

Logistic regression works the same way. The loss function changes - it's cross-entropy instead of squared error - but the penalty term is identical. Most machine learning libraries apply L2 regularization to logistic regression by default, which is why you'll see a parameter called C in scikit-learn. It's the inverse of λ, so a smaller C means stronger regularization.

Neural networks use a couple of different approaches:

• Weight decay: L2 regularization applied to the network's weights - it works the same way, just at a much larger scale
• Dropout: During training, it randomly disables a fraction of neurons in each pass, which prevents the network from relying too heavily on any single path through the layers.

Both reduce overfitting, but through different means.

Tree-based models don't use loss penalties at all. Instead, they control complexity through pruning - limiting how deep a tree can grow, or removing branches that don't improve predictions enough to justify their existence. Hyperparameters like max_depth and min_samples_split in scikit-learn are regularization parameters, even if they're not called that.

Regularization and the Bias-Variance Tradeoff

Regularization is all about compromises.

When you add a penalty term, you're restricting what the model can do. It can no longer fit the training data as closely as it wants. That constraint introduces bias - the model makes slightly wrong assumptions by design, because you've told it to stay simple.

But that same constraint reduces variance. A model that can't fit every data point is less sensitive to the specific samples it was trained on. When you train it on a slightly different dataset, you'll get a similar result. That stability is really what you want, so your model doesn’t fail in production.

Without regularization, you get a highly flexible model that has low bias (makes few assumptions and fits the training data well) and high variance (small changes in the training data produce very different models, which means it can't be trusted on new data).

Regularization is all about shifting the balance. A little more bias in exchange for a lot less variance usually leads to better performance on data the model hasn't seen. That's the compromise, and it's almost always worth making.

Choosing the Regularization Strength

As a machine learning practitioner, you’ll have to set the regularization strength after you pick the regularization type.

That strength is controlled by a hyperparameter - usually called lambda (λ) in math notation, or alpha in scikit-learn. It's the multiplier in front of the penalty term. When you change it, you change how hard the model is pushed toward simplicity.

If you get it wrong in either direction, you’ll have a problem in production:

• Too low: the penalty is too weak to even matter. The model overfits, same as if you had no regularization at all.
• Too high: the penalty is the dominant factor. The model is so constrained that it can't understand real patterns in the data. That's underfitting.

The right value sits somewhere in between, and there's no universal answer. It depends on your data, your model, and how much noise you're dealing with.

The standard way to find it is cross-validation. You split your training data into folds, train the model on each combination of folds, and measure validation performance across a range of alpha values. The value that gives the best average validation score is the one you use.

In scikit-learn, RidgeCV and LassoCV can do this automatically - they run cross-validation over a grid of alpha values and select the best one for you.

from sklearn.linear_model import RidgeCV

model = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0], cv=5)
model.fit(X_train, y_train)

print(model.alpha_)

The printed alpha will show you the best value found by cross-validation. Start with a broad range of values, then narrow it down once you know where the optimal range is.

Conclusion

Regularization is how you stop a model from being too smart for its own good.

It penalizes complexity, which forces the model to find solutions that generalize rather than just memorize training data. L2 will keep all your features and reduce their influence. L1 will remove irrelevant features. Elastic Net combines both. And across linear models, logistic regression, neural networks, and ensamble models, the same idea shows up in different forms, and it’s not always called “regularization.”

What’s most important is the technique you pick and the strength you set. So, what you should do is experiment. Try different approaches with different parameter values. Don't just pick one and move on.

Your data will tell you what works.

If you want to see more regularization techniques in action, enroll in our Machine Learning Scientist in Python track. It has 85 hours of materials that will get you job-ready.