Course
Supervised Learning with scikit-learn
4 hr
300.5K
Every machine learning project eventually gets to a point where you need labeled data, and labeling it costs more time and money than most teams can afford.
Hiring a team to tag a million images or transcribe a million audio clips can run into six figures. At the same time, the internet is full of text, images, audio, and video that's unlabeled. ML teams have more raw data than they'll ever use, but almost none of it comes with the labels that are needed to train the model.
Self-supervised learning doesn't require the labeling step. It turns the data's own structure into the training information. Instead of paying humans to say what's in an image or what a sentence means, you set up a task the data can already answer on its own - like hiding a word and having the model guess it from the words around it.
In this article, I'll break down how self-supervised learning works, walk through its major techniques, and show how it compares with supervised, unsupervised, and semi-supervised learning.
What Is Self-Supervised Learning?
Self-supervised learning is a way to train models without anyone manually labeling the data first.
Instead of a person tagging each example, the data has its own answer key. The model looks at part of the input, hides another part, and learns by trying to fill in the gap. There is no need for a human to intervene.
For example, you can take a sentence, hide one word, and ask the model to guess it from the words around it. "The engineer pushed the ___ to the main branch" gives the model enough context to guess "code" or "changes" without anyone telling it the right answer. The sentence itself is the label.
Self-supervised learning is in a strange spot between two categories. Its training objective looks like supervised learning - there's a clear input, a clear target, and a loss function measuring how wrong the guess was. But since no human created those targets, people usually group it with unsupervised learning, which also works on raw, unlabeled data. Self-supervised learning borrows the mechanics of one and the data source of the other.
How Does Self-Supervised Learning Work?
The process follows a repeatable pattern, no matter what kind of data you're working with. Here are the steps:
- Start with unlabeled data: Text, images, audio, video - it doesn't matter, as long as you have a lot of it.
- Create a learning task from the data itself: This is called a pretext task. It's not the task you actually care about; it's a made-up problem designed to force the model to learn something useful. Hiding a word and predicting it is one example.
- Train the model to solve that task: The model makes a guess, checks it against the real value, and adjusts.
- Learn useful representations: While solving the pretext task, the model builds an internal understanding of the data's structure, such as grammar, context, shapes, and patterns.
- Adapt those representations to downstream tasks: Once the model has learned solid representations, you point it at the task you actually wanted to solve. This is the downstream task.
The pretext task is just a training exercise. Nobody deploys a model that predicts hidden words for fun - the value comes from the representations it picks up along the way, which transfer to actual problems.
Self-Supervised Learning vs Other Learning Methods
The easiest way to understand self-supervised learning is to see where it overlaps with and differs from the other training paradigms.
Self-supervised vs. supervised learning
Supervised learning needs a labeled example for every input, such as an image tagged "dog," or a review tagged "negative." Someone has to create those labels by hand, which means cost, time, and a ceiling on how much data you can realistically use.
With self-supervised learning, the model still gets a target to predict, but the target comes from the data itself - a hidden word or the next token in a sequence. No human annotation is needed to generate the training signal.
Self-supervised vs. unsupervised Learning
Both methods work on unlabeled data, so people often confuse them. The difference is in the objective.
Unsupervised learning looks for structure without a specific prediction target, such as clustering similar customers or reducing a dataset's dimensions. Self-supervised learning builds an explicit prediction task out of the data, then trains the model to solve it. That prediction objective gives self-supervised learning its supervised-style training loop, even though the labels are self-generated.
Self-supervised vs. semi-supervised learning
Semi-supervised learning deliberately mixes a small labeled dataset with a larger unlabeled one, using both at the same time to improve the model.
Self-supervised pretraining doesn't need any labels at all to get started. It can pretrain entirely on unlabeled data, then optionally hand off to a labeled dataset later for fine-tuning. The labels, if they show up, come after pretraining.
| Labels required | Learning signal source | Typical use | |
|---|---|---|---|
| Supervised | Yes, for every example | Human-created labels | Training a model directly on the target task |
| Unsupervised | No | Structure in the data (clusters, patterns) | Grouping, dimensionality reduction |
| Semi-supervised | Some, mixed with unlabeled data | Labeled examples plus patterns in unlabeled data | Boosting accuracy when there isn't much labeled data |
| Self-supervised | No, for pretraining | A pretext task built from the data itself | Pretraining before fine-tuning on a downstream task |
Self-supervised learning compared to other methods
Types of Self-Supervised Learning
Self-supervised methods fall into a couple of broad families, each built around a different kind of pretext task.
Contrastive learning
Contrastive learning trains a model to tell related examples apart from unrelated ones. It pulls the representations of similar examples closer together and pushes dissimilar ones further apart, without ever using a manual label to define "similar."
Two augmented versions of the same image count as related. An image and its matching caption count as related too. SimCLR applies this idea within a single modality, comparing different crops or transformations of the same image. CLIP applies it across modalities, aligning images with the text that describes them.
Masked modeling
Masked modeling hides part of the input and trains the model to reconstruct it. It's the same idea as the fill-in-the-blank example from earlier, just applied more broadly.
In language, this shows up as masked language modeling, which includes hiding a handful of words in a sentence and predicting them from the rest. In vision, it shows up as masked image modeling, which includes hiding patches of an image and asking the model to reconstruct the missing pixels or their underlying representation.
Autoregressive learning
Autoregressive learning predicts the next element in a sequence using everything that came before it. Given the start of a sentence, the model predicts the next word. Given that word, it predicts the one after, and so on.
This is the objective behind most modern language model pretraining. It's a natural fit for text, since language already comes in a left-to-right sequence, and it scales cleanly to huge text corpora without any extra setup.
Self-distillation
Self-distillation trains a model using its own predictions as the target, rather than a fixed answer hidden in the data.
One common setup runs two versions of the same network - a "student" and a "teacher" - on different views of the same input. The student learns to match the teacher's output, and the teacher gets updated as a slow-moving average of the student. DINO and BYOL are representative approaches here. There's no need for negative examples or masked inputs, as the model bootstraps its own supervision from consistency across views.
Self-Supervised Learning in Large Language Models
Self-supervision is the entire pretraining stage for large language models. Every major LLM you've used has learned language structure, facts, and reasoning patterns without a single human-written label, purely by solving self-supervised pretext tasks on raw text.
Next-token prediction
Most modern LLMs pretrain on next-token prediction: given a sequence of tokens, predict the one that comes next. The model reads everything before a position and guesses what follows, checks itself against the actual next token, and adjusts.
If you run that over trillions of tokens scraped from websites and code repositories, the model builds a working model of grammar, facts, and even reasoning patterns along the way. This is the same autoregressive objective covered earlier, just applied at a scale.
Masked language modeling
Before autoregressive pretraining took over, models like BERT used a different pretext task. Instead of predicting what comes next, BERT hides random tokens throughout a sentence and predicts them using context from both sides.
That bidirectional context makes masked language models good at understanding tasks such as classification and question answering, even though they don't generate text the way next-token prediction models do.
Learning from large text corpora
None of this works without volume. Pretraining gets data from web pages, books, code repositories, and forums, filtered and deduplicated into corpora containing billions or trillions of tokens. The bigger and more varied the corpus, the more general the resulting representations become.
Pretraining is where self-supervision does its work, but it's not the last stage. Fine-tuning on human-written instruction examples teaches it to follow directions, and preference optimization methods like RLHF or DPO use ranked human preferences to shape its responses. Both of those later stages need labeled data. Self-supervised pretraining is the only stage that doesn't.
Self-Supervised Learning in Computer Vision
Computer vision is the perfect area that doesn't have enough labeled data. Bounding boxes and segmentation masks take far longer to produce than a single tag on a piece of text, which made self-supervised methods especially valuable here.
Contrastive image learning applies the same pull-together, push-apart idea covered earlier, which includes taking two augmented versions of the same photo (a crop or a flip) and training the model to recognize them as related, while treating other images in the batch as unrelated. SimCLR is a well-known example of this approach applied to images.
Masked image modeling takes the fill-in-the-blank idea from language and applies it to pixels. The idea is to hide patches of an image and train the model to reconstruct the missing content, either as raw pixels or as a learned feature representation. This is the objective behind MAE (Masked Autoencoders).
Image-text alignment trains a model to match images with the captions that describe them, pulling matching pairs together in a shared embedding space and separating mismatched ones. CLIP is the best-known example, and it's part of why models trained this way support zero-shot classification - they can recognize categories they were never explicitly trained on, just by comparing an image against a text description.
Self-Supervised Learning Beyond Text and Images
The pretext task always follows the shape of the data, so wherever you have a lot of unlabeled data, you can usually design a self-supervised task around it.
In speech and audio, models predict masked audio segments or contrast different clips the same way vision models contrast image crops, learning representations useful for tasks like speech recognition without needing transcribed training data upfront.
In video, the pretext task can involve predicting future frames or matching clips that come from the same source video, giving the model a sense of motion and temporal structure that a single image can't provide.
Multimodal models combine two or more of these ideas at once, aligning text, images, audio, and video within a shared representation space, similar to how CLIP aligns images and text, just extended across more types of data.
Self-supervision even shows up in scientific and sensor data - things like genomic sequences or time-series sensor readings - where masking or predicting parts of the signal lets a model learn structure from measurements that were never meant to be "labeled" in the first place.
How Self-Supervised Models Are Used Downstream
Pretraining produces a model that's good at solving a pretext task. Nobody actually cares about this, and what's really important is what you do with it next.
Fine-tuning
Fine-tuning takes the pretrained model and continues training all of its weights, this time on labeled data for the task you actually want. Since the model already understands the structure of the domain, fine-tuning usually needs far less labeled data than training a model from scratch would.
Linear probing
Linear probing freezes the pretrained model completely and trains only a single linear layer on top of its output. If a simple linear classifier can separate classes using those frozen features, the pretrained representations captured real structure. This makes linear probing a common way to test representation quality on its own, separate from how well the full model can be fine-tuned.
Feature extraction
Feature extraction works the same way as linear probing, but without the linear part. You take the frozen representations and feed them into any downstream algorithm you want. The pretrained model's only job is generating good embeddings, everything downstream of that is up to you.
Zero-shot and few-shot use
Some representations are good enough to use with no downstream training at all. CLIP does this by embedding an image and a set of candidate text labels into the same space, then picking the label closest to the image. Few-shot learning takes the same idea and adds a small number of labeled examples, which is often enough because the representations already carry most of the structure a model needs.
With self-supervised learning, you get reusable representations. You pretrain once, then fine-tune, probe, extract features from, or query that same core across as many downstream tasks as you need. Most of the expensive work happens a single time during pretraining, not once per task.
Self-Supervised Learning in Python
Here's a lightweight example using a pretrained model, not a self-supervised system built from the ground up - just so you can get the sense for the workflow.
all-MiniLM-L6-v2, available through the sentence-transformers package, was pretrained with a masked language modeling objective and then further trained with a contrastive objective on sentence pairs. That's the self-supervised part already done for you - you're just putting the resulting representations to work.
For starters, make sure you install the dependencies:
pip install sentence-transformers scikit-learnNow onto the code:
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
# Create unlabeled data
support_tickets = [
"My order arrived damaged and I need a replacement.",
"The app crashes every time I try to upload a photo.",
"I love how fast the checkout process is now.",
"Can you tell me when my refund will be processed?",
"The new dashboard layout is so much easier to use.",
"I can't log in, it keeps saying my password is wrong.",
]
# Load a model pretrained with a self-supervised objective
model = SentenceTransformer("all-MiniLM-L6-v2")
# Obtain learned representations
embeddings = model.encode(support_tickets)
print(embeddings.shape)
Embeddings shape
The .encode() method runs each ticket through the pretrained model and returns a dense vector that represents its meaning. From there, a downstream task only needs a handful of labeled examples, since the embeddings already carry most of the structure:
# Use the representations for a downstream task, 1 = complaint, 0 = positive feedback
labels = [1, 1, 0, 1, 0, 1]
clf = LogisticRegression()
clf.fit(embeddings, labels)
new_ticket = ["The delivery was late and no one responded to my email."]
new_embedding = model.encode(new_ticket)
print(clf.predict(new_embedding))
Logistic regression prediction
Six labeled tickets and a LogisticRegression classifier are enough here because the representations, not the classifier, are doing most of the work.
Advantages and Limitations of Self-Supervised Learning
Let me now go through pros and cons of self-supervised learning.
Advantages
- Reduces dependence on manually labeled data: The pretext task generates its own targets, so labeling budgets is no longer a hard limit for how much data you can train on
- Makes use of very large datasets: Web-scale text and audio collections become usable for training, not just the small labeled subsets teams could afford to annotate
- Produces transferable representations: A single pretrained model can support fine-tuning, linear probing, feature extraction, and zero-shot use across a couple of different downstream tasks
- Enables large-scale pretraining: This is what makes today's foundation models possible in the first place.
Limitations
- Substantial compute requirements: Pretraining on internet-scale data takes serious hardware and time, well out of reach for most individual teams
- Pretext task design matters: A poorly chosen pretext task produces representations that don't transfer well, no matter how much data you throw at it
- Learned representations can inherit biases: Whatever patterns, gaps, or skew exist in the training data show up in the representations too, since nothing filters them out during pretraining
- Strong pretraining performance doesn't guarantee downstream success: A model that solves its pretext task well can still underperform on a specific downstream task it was never directly optimized for
Why Self-Supervised Learning Matters for Modern AI
Self-supervised learning is the reason foundation models are possible at all.
Pretraining at the scale modern models require would be out of reach if it depended on manual labels. Nobody labels a trillion tokens of text or a billion images by hand. Self-supervision made it practical to pretrain on internet-scale collections, then reuse that single model across a wide range of downstream tasks instead of training a separate model for each one.
That change from many task-specific models to one general-purpose pretrained model adapted to many uses is the direction most of modern AI has moved in. Self-supervised learning is the mechanism that made the move possible.
Conclusion
Self-supervised learning generates its own supervision from the structure of unlabeled data instead of labels a human wrote. The typical pattern is to hide a word or predict the next token. The data answers its own questions, and the model learns by trying to get those answers right.
The major approaches all build on that same idea in different ways. Contrastive learning pulls related examples together and pushes unrelated ones apart. Masked modeling hides part of the input and trains the model to reconstruct it. Autoregressive learning predicts the next element in a sequence from everything before it. The mechanics are different, but the underlying principle of letting the data's own structure supply the training signal is the same.
That principle is why self-supervision is behind most of the language, vision, and multimodal models you already use. Next-token prediction pretrains today's LLMs, contrastive and masked objectives pretrain vision models, and image-text alignment pretrains the multimodal models that tie the two together.
If you want to learn to create an LLM from scratch, enroll in our Developing Large Language Models track to see PyTorch, Hugging Face, and the latest NLP techniques in action.
What is self-supervised learning?
How is self-supervised learning different from unsupervised learning?
Why does self-supervised learning matter for modern AI?
What's the difference between a pretext task and a downstream task?
Can self-supervised learning work with zero labeled data?
Topics
