Hyperparameter Tuning: Methods, Best Practices, and Examples

The accuracy of your model isn't bad because you chose the wrong algorithm - it's because you never tuned it.

Most people train a model with default settings, see the accuracy settle around 80-ish percent, and say the algorithm isn't suited for their task. But the algorithm isn't the problem. Every ML model has parameters that control how it learns, and those parameters are set before training even starts. If you get them right, you can get out 10-20% more accuracy from the same model on the same data.

These parameters are called hyperparameters, and choosing them well will separate a model that's okay from the one that always seems to get it right. Hyperparameter tuning is the process of searching for the combination that gives you the best performance.

In this article, I'll walk you through what hyperparameter tuning is, the search strategies that work best, how to evaluate results without fooling yourself, and how to run it all in Python and R.

If you're new to data science and machine learning, check the Machine Learning Fundamentals in Python course before working on more advanced topics.

What Is Hyperparameter Tuning?

Hyperparameter tuning is the process of finding the settings that make a model learn best.

Every ML model has two sorts of numbers. Some are learned from the data during training, like the weights in a neural network or the split points in a decision tree. Others are set before training even begins, and those are the hyperparameters. They control how the model learns - how fast, how deep, how much it should generalize or memorize.

Default hyperparameters exist to get you a model that runs. They're not built to give you the best model for your specific dataset. A default learning rate that works for image classification logically won't perform well on tabular data. A default tree depth that's fine for a clean dataset will overfit on a noisy one. You get the idea.

The process of tuning searches through different combinations of hyperparameter values, training a model for each, and keeping the one that performs best on your validation data. 

What you can get from this is often better than switching to a better but untuned algorithm.

Hyperparameters vs Model Parameters

The difference comes down to who sets them and when.

Model parameters are learned by the algorithm during training. You never set them yourself. The training process adjusts them by looking at the data and minimizing a loss function. For example:

• Weights and biases in a neural network
• Coefficients in a linear regression
• Split thresholds in a decision tree
• Support vectors in an SVM

Hyperparameters are chosen by you before training starts. They control how the learning process works, but the algorithm won't adjust them in any way. If you don't set them, the library uses its defaults. For example:

• Learning rate in gradient descent
• Number of trees in a random forest
• Regularization strength in logistic regression
• Kernel type in an SVM

Here's a side-by-side comparison:

Model parameters answer what the model learned. Hyperparameters answer how it learned.

How Hyperparameter Tuning Works

Tuning is a loop.

You start by picking the hyperparameters you want to search over and defining a range of values for each. Then you train a model with one combination, measure how it performs on validation data, and use that result to decide what to try next.

The loop looks like this:

1. Select hyperparameters to tune and pick a range of values for each one.
2. Train the model with a specific combination of values.
3. Evaluate performance on a validation set using a metric that matches your task.
4. Update the hyperparameters based on what you learned - either by moving to the next combination in a predefined list or by using the last result to guide the next choice.
5. Repeat until the performance stops improving or you run out of compute budget.

The choice of how you update between iterations separates one search method from another. Manual tuning updates by intuition. Grid search updates by walking through a predefined list. Bayesian methods update by building a model of which combinations are likely to work.

Let me dive into these methods next.

Hyperparameter Search Methods

The search method you choose controls how much compute you go through and how likely you are to find good hyperparameters.

Manual search

Manual search is the simplest method. You pick a combination of hyperparameters, train the model, look at the result, and adjust based on what you saw.

It works when you have a small model, a few hyperparameters, and prior experience with the algorithm. If you already know that XGBoost usually likes a learning rate around 0.05 and a tree depth of 6, you can start there and adjust from what you observe.

Manual search doesn't work when the search space grows. Three hyperparameters with five reasonable values each already give you 125 combinations, and no human tunes 125 models by hand.

Grid search

Grid search checks every combination in a predefined list of values.

You define a grid - for example, learning rates of [0.01, 0.05, 0.1] and tree depths of [3, 5, 7] - and grid search trains a model for each of the nine combinations. It's exhaustive and reproducible.

The problem is the compute cost. 

Adding one more hyperparameter with five values multiplies your training runs by five. Grid search also wastes effort on values that don't matter - if tree depth barely affects your score, grid search still trains a model for every depth you listed.

Grid search is a good default for small search spaces with two or three hyperparameters.

Random search

Random search samples random combinations from a range instead of walking through a fixed grid.

You define a distribution for each hyperparameter and random search draws combinations from those distributions. It's more efficient than grid search because it doesn't waste runs on values that don't matter - if only one out of three hyperparameters affects performance, random search still explores the important one across many different values.

Grid search versus random search

Random search is the better default for anything with more than two hyperparameters.

Bayesian optimization

Bayesian optimization uses the results from previous trials to decide what to try next.

It builds a probabilistic model of how hyperparameter combinations map to performance, then picks the next combination that either has the highest expected score or the highest uncertainty. This is the exploration-exploitation tradeoff, meaning you either explore new regions that might be good, or exploit regions you already know are good.

Bayesian methods find good hyperparameters in far fewer trials than random or grid search. The tradeoff is that each trial takes a bit longer because the algorithm has to fit its internal model before choosing the next point. Libraries like Optuna and Hyperopt do this for you.

Use Bayesian optimization when training is expensive and you can only afford a small number of runs.

Successive halving and Hyperband

Successive halving and Hyperband reduce compute time by stopping bad runs early.

Successive halving starts many hyperparameter combinations with a small training budget, throws out the worst half, doubles the budget for the survivors, and repeats. 

Hyperband is a wrapper around successive halving that runs it multiple times with different budget splits, so you don't have to guess the right starting configuration.

Both methods are useful when training is slow and you have a large search space. They work best when early performance is a decent signal for final performance, which is usually true for neural networks and gradient boosting.

Here's how the methods compare:

Evaluating Hyperparameter Performance

Tuning won't get you far if you don't choose a correct range of values.

If your evaluation is off, you'll pick the wrong hyperparameters. There are four things to get right - how you split the data, how you measure performance, which metric you optimize, and how you keep the test set clean.

Train, validation, and test splits

Split your data into three parts before tuning.

The training set is what the model learns from. The validation set is what you use to score each hyperparameter combination during tuning. The test set is what you use once, at the very end, to measure how the final model performs on unseen data.

If you tune against the test set, you'll get an accuracy number that looks great and a model that underperforms in production. Every tuning decision you make against the test set leaks information about it into your model.

A common split is 60/20/20 for medium-sized datasets or 80/10/10 for larger ones.

Cross-validation

Cross-validation is a better way to score hyperparameters when your validation set is small.

You split the training data into k folds - usually 5 or 10. For each hyperparameter combination, you train on k-1 folds and validate on the remaining one, then rotate. The final score is the average across all folds.

Cross-validation gives you a more stable estimate of how well a hyperparameter combination generalizes, because you're not relying on a single lucky or unlucky validation split. The cost is that you train k models per combination instead of one.

Use cross-validation when your dataset is small enough that a single validation split isn't reliable, and when you can afford the extra training cost.

Selecting evaluation metrics

The metric you optimize should match what you actually care about.

Accuracy is fine for balanced classification but doesn't make sense for imbalanced problems. For example, a fraud detection model that predicts "not fraud" for everything can get 99% accuracy while missing every case. For imbalanced classification, use precision, recall, F1, or ROC-AUC. For regression, use RMSE, MAE, or R squared depending if outliers matter.

Just pick one primary metric before you start tuning and stick with it.

Avoiding data leakage

Data leakage happens when information from your validation or test set gets into training.

Here are some common ways it happens:

• Scaling or normalizing the full dataset before splitting, so validation statistics leak into training
• Feature engineering that uses target values from the whole dataset
• Tuning against the test set instead of a separate validation set
• Using time-series data without respecting time order in the split

Fit all preprocessing steps on the training set only, then apply them to validation and test. In scikit-learn, this is what Pipeline is for - it wraps preprocessing and modeling into one object that respects your splits.

Implementing Hyperparameter Tuning

Most ML libraries include tools that do the search and the cross-validation, so you don't have to implement anything from scratch.

Hyperparameter tuning in Python

Scikit-learn comes with two tuning classes that cover most use cases - GridSearchCV for grid search and RandomizedSearchCV for random search. Both wrap around any scikit-learn estimator, run cross-validation for each combination, and give you the best model at the end.

GridSearchCV takes an estimator, a dictionary of hyperparameter values, and the number of cross-validation folds. It trains a model for every combination and keeps the one with the best score.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, random_state=42)

param_grid = {
    "n_estimators": [100, 200, 500],
    "max_depth": [5, 10, 20],
    "min_samples_leaf": [1, 5, 10],
}

grid = GridSearchCV(RandomForestClassifier(random_state=42), param_grid, cv=5)
grid.fit(X, y)

print(grid.best_params_)
print(grid.best_score_)

GridSearchCV results in Python

RandomizedSearchCV uses the same interface but samples from distributions instead of going through a fixed list. This lets you cover a wider range with a fixed number of trials.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    "n_estimators": randint(100, 500),
    "max_depth": randint(3, 20),
    "min_samples_leaf": randint(1, 10),
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_dist,
    n_iter=30,
    cv=5,
    random_state=42,
)
random_search.fit(X, y)

print(random_search.best_params_)

RandomizedSearchCV results in Python

For Bayesian optimization, Optuna is the standard choice. It picks the next combination based on previous results, so it converges faster than random search when training is expensive.

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 500),
        "max_depth": trial.suggest_int("max_depth", 3, 20),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 10),
    }
    model = RandomForestClassifier(**params, random_state=42)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=30)

print(study.best_params)

Optuna results in Python

The trial.suggest_* methods define the search space, and Optuna decides which combination to try next based on the scores it has seen so far.

Hyperparameter tuning in R

R has two great options - caret and tidymodels. Both do the same - define a grid, run cross-validation, return the best model - but tidymodels is the modern default.

With caret, you pass a tuning grid to the train() function and specify the resampling method.

library(caret)

data <- iris
ctrl <- trainControl(method = "cv", number = 5)

grid <- expand.grid(
  mtry = c(2, 3, 4)
)

model <- train(
  Species ~ .,
  data = data,
  method = "rf",
  trControl = ctrl,
  tuneGrid = grid
)

print(model$bestTune)

Caret results in R

tidymodels splits the workflow into clear steps - a model spec, a recipe for preprocessing, and a tuning grid. It's more verbose but in practice is more scalable.

library(tidymodels)

data <- iris
split <- initial_split(data, prop = 0.8)
train_data <- training(split)

rf_spec <- rand_forest(
  mtry = tune(),
  trees = tune(),
  min_n = tune()
) |>
  set_engine("ranger") |>
  set_mode("classification")

grid <- grid_regular(
  mtry(range = c(2, 4)),
  trees(range = c(100, 500)),
  min_n(range = c(1, 10)),
  levels = 3
)

folds <- vfold_cv(train_data, v = 5)

results <- tune_grid(
  rf_spec,
  Species ~ .,
  resamples = folds,
  grid = grid
)

show_best(results, metric = "accuracy")

Tidymodels results in R

The workflow is exactly the same as in Python - you define what to search over, define how to score it, and let the library run the loop.

Which Hyperparameters Should You Tune?

Not every hyperparameter is worth tuning because usually only a few control most of the performance. Here are the ones that usually matter first, by model family.

Decision trees and random forests

You should always tune these three:

• Maximum depth controls how far each tree can grow. Shallow trees underfit, deep trees overfit. This is usually the first thing to tune.
• Minimum samples per leaf sets how many training points a leaf node needs. Higher values force the tree to generalize instead of memorizing individual rows.
• Number of trees in a random forest. More trees give you a more stable model but with diminishing returns past a few hundred. Tune this last.

Gradient boosting models

Gradient boosting (XGBoost, LightGBM, CatBoost) is more sensitive to hyperparameters than random forests. Pay attention to these:

• Learning rate controls how much each new tree corrects the previous ones. Lower rates are more accurate but need more trees. This is the single most important hyperparameter to tune.
• Number of estimators is the number of boosting rounds. Tie it to the learning rate - if you lower one, raise the other.
• Tree depth controls how complex each individual tree is. Boosting usually likes shallower trees than random forests, typically 3 to 8.

Support vector machines

With SVMs, the choice of hyperparameters matters a lot, arguably more than with other models:

• C controls the tradeoff between a wide margin and correctly classifying training points. Higher C means less regularization.
• Gamma controls how far the influence of a single training point reaches. Low gamma means far, high gamma means close.
• Kernel determines what shape of decision boundary the SVM can learn. Start with rbf for non-linear problems and linear for high-dimensional sparse data.

Neural networks

Neural networks have the largest search space of any common model family. Tune these first:

• Learning rate is the most important hyperparameter in deep learning. Too high and training diverges, too low and it never converges. Try values across orders of magnitude such as 1e-4, 1e-3, 1e-2.
• Batch size affects both training speed and final accuracy. Larger batches train faster per epoch but often generalize worse.
• Optimizer choice (Adam, SGD with momentum, AdamW) changes how the network updates its weights. Adam is a good default.
• Epochs is how many times the model sees the full training set. Use early stopping instead of tuning this by hand.

Common Challenges in Hyperparameter Tuning

Tuning is expensive, especially when you want to test multiple values per parameter and have many parameters to tune. Here are some common challenges to know.

Compute cost is the biggest constraint. Every combination you try means training a full model, and cross-validation multiplies that by the number of folds. A grid of 100 combinations with 5-fold CV is 500 model fits. If each fit takes 10 minutes, that's over three days.

Large search spaces grow multiplicatively. Five hyperparameters with five values each is 3,125 combinations. This is where grid search is no longer an option and is replaced with random search or Bayesian methods.

Diminishing returns are visible earlier than most people expect. The first few tuning runs usually give you the biggest gains. The next 50 might get out another percent of accuracy. At some point the extra compute would be better spent on getting more data or fixing your features.

The accuracy-time tradeoff matters more in production than in development. A model that scores 0.5% higher but takes twice as long to train might not be worth it if you retrain daily. You should decide up front how much accuracy is worth how much compute.

Best Practices for Hyperparameter Tuning

Here are some good habits to follow when tuning hyperparameters of a machine learning model:

• Tune the most influential hyperparameters first: Learning rate for boosting and neural networks. Max depth for trees. C and gamma for SVMs. Get these in the right range before addressing anything else.
• Start with random search for large search spaces: Grid search wastes compute on unimportant hyperparameters. Random search covers the space better, and it's the right choice for anything with more than two or three hyperparameters.
• Use cross-validation: A single validation split can be lucky or unlucky. Five folds give you a much more honest estimate of how a combination generalizes.
• Define realistic search ranges: If your learning rate range is [0.001, 100], you'll waste time on values that never work. Start with ranges that are known to be reasonable for your algorithm, then narrow them based on what you see.
• Track your experiments: Log the hyperparameter values, the score, and the seed for every run. Tools like MLflow, Weights & Biases, or even CSV file all work. Without this, you can't reproduce your results.
• Set a compute budget: Decide how many trials you can afford before you start, not after. This forces you to pick a search method that fits and stops you from tuning longer than necessary.

Common Mistakes in Hyperparameter Tuning

Most tuning fails are due to mistakes that are easy to avoid. Here are a couple:

• Confusing hyperparameters with model parameters. Weights and coefficients are learned during training. You don't set them by hand and you don't tune them. If you're trying to tune something the algorithm should be learning, you've mixed up the two.
• Tuning against the test set. Every decision you make based on the test score leaks information about it into your model. The number you get at the end will be optimistic, and your production model will underperform. Use a separate validation set or cross-validation.
• Searching unnecessarily large parameter grids. A grid with seven hyperparameters and ten values each has ten million combinations. You can't search that. Cut it down before you start by picking the two or three hyperparameters that matter most and give them realistic ranges.
• Optimizing the wrong metric. Accuracy on an imbalanced dataset will give you a model that predicts the majority class every time. Pick a metric that reflects what you actually care about, and pick it before you start tuning.
• Changing too many hyperparameters at once during manual tuning. If you adjust the learning rate and batch size in the same step, you won't know which one caused the change. Move one at a time when you're tuning by hand.

Hyperparameter Tuning vs Feature Engineering

Both improve your model, but they work on different parts of the problem.

Feature engineering improves the input data. You create new features from existing ones, encode categorical variables, handle missing values, and remove noise. Better features give the model more useful signals to learn from.

Hyperparameter tuning improves how the model learns from whatever features you give it. It changes the algorithm's behavior on the data instead of changing the data.

The two are complementary. Feature engineering usually gives you the largest gains early in a project, when the data is still raw and the model is missing obvious signals. Hyperparameter tuning gives you smaller but consistent gains once your features are good enough.

If your model is stuck and you're not sure which one to try first, look at your errors. If the model is missing patterns that seem obvious from the data, you have a feature problem. If it's overfitting or underfitting, you have a tuning problem.

Why Hyperparameter Tuning Matters in Machine Learning

Tuning is a standard step in any ML workflow.

It often gives you larger gains than switching algorithms. For example, a well-tuned random forest will often beat a poorly-tuned gradient boosting model on the same data, even though gradient boosting is the more sophisticated algorithm. 

Tuning also helps with generalization. Hyperparameters like max depth and dropout rate directly control how much the model can overfit. Getting these right is what makes the difference between a model that memorizes the training set and one that performs well on new data.

In production, tuning turns a proof-of-concept into a system that is reliable. 

The default settings get you started, but these are not the settings that will keep the model competitive against retraining and business requirements. Every serious ML team treats tuning as part of the process.

Conclusion

Hyperparameter tuning is an iterative loop of searching, evaluating, and narrowing in on what works for your specific data.

The best results come from combining three things - a search method that fits your compute budget, validation that gives you a score that's relevant, and experiment tracking that lets you go back and see what you tried.

Start with random search or grid search. Move to Bayesian optimization when training gets expensive enough that trial count actually matters. Either way, tune the hyperparameters that are relevant for your model, keep your test set clean, and log every run.

If you're considering a career in machine learning, enroll in our Machine Learning Engineer track to get job-ready in 2026.