Generative vs Discriminative Models: Differences & Use Cases

What Are Discriminative Models?

Given a training dataset consisting of data points, D, and their associated labels, L, a discriminative model learns the conditional probability distribution P(D | L). It then uses this conditional probability distribution to predict the class of new data points. 

Discriminative model workflow. Created with napkin.ai

Discriminative models are generally used to solve classification problems. The following examples demonstrate their use cases. 

K-nearest neighbors

K-nearest neighbors (KNN) is one of the older machine learning models. It is based on the premise that, given a distribution of data points, similar items are located in proximity. 

KNN models are nonparametric, with no parameters like regression coefficients. They are used for both classification and regression problems. 

An input data point's category is the same as its k nearest neighbors. The predicted value of a datapoint is the average value of its k nearest neighbors. k (the number of nearest neighbors to consider) can be considered a hyperparameter of the model. It is used to tune the model's behavior, but it doesn’t directly affect the model’s output.

Logistic regression

Logistic regression, like linear regression, tries to predict the value of a dependent variable based on one or more independent variables. In linear regression, the dependent variable takes on continuous values. In logistic regression, the dependent variable takes discrete values, such as:

• True and false (binary logistic regression)
• A list of categories (multinomial logistic regression)
• A list of ordered values, like a letter-based grading system - A, B, C, D, E, F (ordinal logistic regression)

Linear regression predicts numerical values. In logistic regression, the predicted quantity is the logarithm of the odds ratio. 

For an event A, with probability P(A), the ratio of odds is P(A) / (1 - P(A)). Using the logarithm (of the odds ratio) leads to smoother and faster convergence during the training process.  Given their ability to segregate inputs into classes, logistic models are used for discriminative purposes. 

For a more hands-on introduction to this topic, refer to the guide on logistic regression in Python.

You can also implement logistic regression using R, a statistics-oriented programming language, as explained in the tutorial on logistic regression using R.

Support vector machines (SVM)

Support vector machines determine the optimal line that separates data points of different classes. In the 2-dimensional X-Y plane, given a collection of data points of two different categories, the SVM predicts a line that (ideally) cleanly separates points of one category from the other. 

This line is the decision boundary. It becomes a hyperplane for data with three or more dimensions. Data points (from either category) that lie closest to this imaginary line are called the model's support vectors. These data points are the hardest to classify since their values are close. 

The distance between the separating line and the support vectors is called the margin. SVM training aims to find the decision boundary that maximizes this margin. In practice, data points have more than two dimensions, and the separating line is a higher-dimensional hyperplane. 

SVMs are also used for multi-class classification problems.

To learn more about SVMs, follow this guide on building SVMs using the Python scikit-learn package. Besides Python, you can also use R to implement SVMs, as discussed in this guide on SVMs in R.

Decision trees and random forests

A decision tree consists of several decision nodes organized in a tree-like structure. 

The topmost node is the root. Nodes that lead to final outputs are called leaf nodes or terminal nodes. Intermediate non-leaf nodes are called internal nodes. The output from the root node feeds into the intermediate nodes. Each node's result (output) is either the final output or leads (branches) to another node. 

Each node of the tree splits the dataset along a particular attribute. For example, a decision tree for approving loan applications might have nodes to segregate applications by their net income. 

The training process determines the appropriate threshold value for each decision. For example, applications with net income below a certain amount are outright rejected. The rest are processed further in subsequent nodes, which consider other attributes, like net wealth. 

For a hands-on guide to decision trees, follow the tutorial on building decision trees using Python, or the comple course on machine learning with tree-based models in Python.

The main drawback of decision trees is overfitting, leading to out-of-sample data problems. Decision forests try to solve this problem. A decision forest consists of many trees. Unlike standalone decision trees, which must consider the entire feature set, each tree in a forest only finds a random subset of the feature set. This randomness helps tackle the variance in noisy datasets. 

The forest output is obtained by combining, for example, by averaging, the output of individual trees.

To learn how to implement random forests, refer to the guide on using scikit-learn to build random forest classifiers.

Neural networks

A neural network consists of groups of neurons. Each neuron implements a linear function that multiplies the neuron's weights with the input vector. 

A non-linear activation function follows the linear function. The activation function decides the output of each neuron based on the output of the linear function. Thus, a simple neural network can conceptually be seen as a series of linear equations filtered by non-linear activations.

The input layer of a neural network multiplies the input (represented as a vector) by a set of weights and activation functions. This output is passed to the next layer, which performs a similar operation. 

Hidden layers are between the input and output layers. The last hidden layer feeds into the output layer. Complex problems involve using neural networks with many hidden layers, which are called deep neural networks (DNNs).

In a classification problem, a common approach is to have as many output neurons as the number of classes. The predicted class corresponds to the neuron with the highest value. In a regression problem, a single output neuron contains the predicted output. The neurons' linear relationships model the linear regression equation. 

For a more in-depth understanding of the topic, check out the blog post on neural networks.

Differences Between Generative and Discriminative Models

Some tasks, like classification, can be solved using either model type. In general, however, discriminative and generative models often have unique use cases, as these models adopt different mathematical approaches. Understanding these differences and how they affect the suitability of either category of models to various problems is necessary. 

Approach to modeling

Generative models predict the next value in a sequence or generate an image given a text (or vice versa). To do such tasks, the model has to learn which output to generate given different inputs. The model uses the joint probability distribution of the input and the output. 

To understand the underlying mathematics, let’s start with a simple example. The behavior of a single random variable is described by its probability density function (PDF). 

The PDF of the random variable X can be used to determine the probability of X at different values. For example, if the PDF of X is f(x), the probability that X lies between A and B is given by: 

The probability of X over the entire range is 1. This is expressed as : 

The above expression can also be written as: 

For a joint PDF of two variables, X and Y, the integral over the entire range is 1:

The joint probability distribution maps the entire probability space of both X and Y. To evaluate the probability that the joint probability P(X, Y) falls in a region G, integrate the joint PDF over G:

Discriminative models, instead, focus only on the conditional probability distribution. Generative models, if needed, estimate the conditional probability using the marginal probabilities. 

Given a joint probability distribution f_XY(x,y), the marginal probability of the random variable Y at the value y (for all values of X) is given as:

Similar to Bayes rule, the conditional PDF of X is then expressed as:

The above expression shows the integral version of the conditional probability, which is more commonly written as:

Similarly,

In the two conditional probability formulae above:

• The left-hand side (LHS) is the conditional probability (the conditionals)
• The numerator of the right-hand side (RHS) is the joint probability
• The denominator of the RHS is the marginal probability. 

To compute a conditional probability, generative models follow two steps:

• Estimating the joint PDF and the marginals. 
• Evaluate the expression on the right-hand side of the equation above.

On the other hand, discriminative models follow a single (and simpler) step:

• Directly estimate the conditional probability (the left-hand side of the equation) based on the training data. 

Flexibility and complexity

A generative model has the marginals P(A) and P(B) and the joint PDF P(A, B). Using these, it can evaluate P(A | B) or P(B | A) with equal ease. This is the underlying idea behind naive Bayes classifiers. Thus, generative models can do tasks like:

• Generating new examples using the joint probability distribution. 
• Predict the most relevant label given a new example using the conditional probability. 
• Predicting the most relevant example given a new label using the other conditional probability. 
• Estimating the overall probability of certain events using the marginal probabilities. 

This makes generative models flexible and multipurpose. On the other hand, they inherently involve more complexity during training because:

• The model needs to estimate the joint PDFs and the marginal
• Then, if needed, the model evaluates the conditionals

Discriminative models, on the other hand, are concerned only with conditionals. Based on the training dataset, it is possible to estimate the conditional probabilities directly without estimating the joint or marginal probabilities. 

Thus, discriminative models are simpler to train. However, a discriminative model that has learned the conditional probability P(A | B) can only perform tasks involving this particular conditional probability. It cannot do anything else. 

Performance in classification tasks

Generative models are flexible enough for both generative and discriminative tasks. During training, the model learns the joint PDF and the marginals. During inference, the model must compute the conditional probability using the joint distribution and the appropriate marginal probability. Hence, inference on discriminative tasks is slower.

Discriminative models, on the other hand, have directly (numerically) learned the conditional probabilities. They estimate the conditional probability in a single step based on the input data during inference. 

Furthermore, because discriminative models are focused only on estimating a single quantity (the conditional probability), they are observed to be more accurate. A joint PDF has more uncertainty built into it than a simple conditional probability. This added uncertainty is reflected in the relatively lower accuracy of generative models for classification tasks. 

Generative vs Discriminative Models: A Summary

Based on the discussion in the previous sections, the table below summarizes the differences between generative and discriminative models. 

When To Use Generative Models

As the name suggests, generative models are uniquely suited to tasks that involve generating new data that fit the patterns in the training data. 

Some common use cases of these models are:

1. Synthetic data generation

For many computer vision models, having a large dataset of training images is essential. These models need to be trained on many variants of the same image or feature. 

Taking pictures of the same object from different angles, backgrounds, or shades of color is not always realistic. Many task-specific image datasets also tend to be limited in size because creating them requires domain-specific skills. For example, you need access to pulmonologists, radiology labs, and consenting patients and hospitals to create an image set of X-rays of a specific lung disease. 

In such cases, the most pragmatic option is to create a small but highly curated and relevant dataset and then synthetically generate new data points similar to the original dataset. 

2. Image and text generation

Generating new images given a text description or generating new text given some input text. The only class of models that can handle such tasks are generative models. 

Image generation models like DALL-E, MidJourney, and Stable Diffusion are routinely used to generate images that suit a specific description and do not infringe existing copyright. 

Similarly, LLMs are often used to generate fictional story plots, marketing slogans and taglines, document summaries, and other similar materials. 

3. Modeling joint probability distributions

Generative models are used for tasks based on the underlying data's joint probability distribution. Modeling the joint distribution of asset returns to predict an investment portfolio's expected risk and return profile is standard in finance and risk management.

For example, an investment manager might want to know the probability of the prices of two different stocks rising or falling at the same time—joint PDFs can answer such questions. 

The financial services industry has been using these statistical methods long before the term “generative models” was widely adopted.

4. Unsupervised classification

Problems like unsupervised classification are uniquely suited to Gaussian mixture models. In these problems, you have an extensive collection of data points but do not know how many or what categories the data points belong to. It is reasonable to expect that the data comes from a combination of distributions, which a GMM can model. 

If the number and list of categories had been known in advance, discriminative methods would have been more appropriate. 

GMMs are also helpful in outlier analysis (anomaly detection, fraud detection, etc.), where the “usual” data, like the behavior of different customer groups, can be modeled as a combination of different distributions. Anomalous data points are those that differ significantly from any of the other patterns. 

5. Sequence modeling

Problems involving sequences are often solved using hidden Markov models. An everyday use case is modeling genome sequences and re-sequencing. 

HMMs are also used in speech recognition, where they help predict the next syllable given the sequence of preceding syllables. Sequence modeling is also standard for logistics applications like package delivery schedules. Similarly, transmission and spread of infectious diseases are often modeled using Markov chains. 

When To Use Discriminative Models

Mathematically, discriminative models are used for applications where only the conditional probability is relevant but not the joint probability. Thus, discriminative models are typically used for classification problems and prediction problems. 

Some examples are:

1. Supervised classification problems

Supervised classification problems, where the classes (categories) are known in advance. This is the quintessential use case for KNNs and SVMs. 

You have an extensive collection of data points, such as customer behavior data, including spending patterns, purchase amounts, buying frequency, return history, and so on. You need to use this information to classify the list of customers into different categories, such as high spenders, bargain hunters, regular customers, non-serious customers, etc. Contrast this to unsupervised classification problems, where, as discussed earlier, GMMs are commonly used.

2. Predictive tasks

In predictive tasks like classification and regression, the priority is often speed and accuracy. In principle, generative models are also capable of solving such problems. 

In particular, generative models are preferred when there are issues like missing data points, when only a limited dataset is available for training, or when you expect to add new categories routinely. 

However, for most standard use cases, generative models suffer from higher error rates and slower inference in predictive tasks because their computation involves the joint probability distribution. Hence, for standard classification and regression problems, where a large and healthy dataset is available for training, discriminative models are the preferred choice due to their better performance characteristics. 

3. Outcome-focused classification

Discriminative models are preferred for tasks that focus only on the classification's outcome and not on modeling the underlying data. 

For example, if you want to categorize audio recordings into their respective languages, you do not need a model that understands the language and the grammar. A speech recognition model would be overkill for a classification problem. It is sufficient to focus on the decision boundary using a discriminative model. So, you can use a simple neural network instead of an LLM or logistic regression instead of a Bayesian model.

4. Multi-step decision making

Multi-step decisions, where the individual steps involve unambiguous choices, are good candidates for decision trees. Decision trees are often used as a preliminary filter to shortlist data points that require deeper analysis. 

For example, consider fraud detection, often solved using complex models like GMMs. Humans almost always investigate high-risk transactions, regardless of the model used. So, an alternative approach is to use a decision tree to flag a list of potentially fraudulent transactions for further manual investigation. 

Decision trees are also used in many business operations. The goal is to decide the right course of action based on predetermined conditions. Such tasks need neither generative models nor the more complex discriminative models like neural networks. 

Example applications of generative vs. discriminative models. Created with napkin.ai

Conclusion

This article explained the fundamental principles and main differences between generative and discriminative models, the two major approaches to machine learning techniques.

Though most machine learning is based on probabilistic methods, generative models are based on the joint probability distribution, while discriminative models use only conditional probabilities. Thus, the two classes of models have different applications and performance characteristics. Considering the availability of a wide variety of models, choosing the right tool for the job becomes essential. 

Beyond a conceptual understanding of model types and their differences, it is more important to build models yourself. Start with the blog post discussing supervised training methods and building a simple logistic regression model.