Skip to main content

Building a Q&A Assistant in Django with MongoDB Auto-Embeddings

Build semantic search in Django with MongoDB auto-embeddings: no embedding code, automatic re-sync when documents change, and plain-text queries.
· 7 min read

Explore with AI

What Auto-Embeddings Simplify

In a typical retrieval-augmented generation (RAG) setup, you generate embeddings when you insert a document, regenerate them when the content changes, and embed every query yourself. Automated Embedding simplifies the process of building semantic search, as MongoDB Atlas handles embedding generation, updates, and querying natively, and Vector Search stores generated vector embeddings on your cluster.

How MongoDB does automatic embedding

When a vector search index of type autoEmbed is created, MongoDB generates embeddings for the chosen field, uses the VoyageAI model, keeps them in sync when there are changes to the document, and embeds the query text at search time with the same model. The generated embeddings are stored asynchronously on the MongoDB cluster in a separate internal database.

In this tutorial, we will build a Q&A Docs Assistant with MongoDB autoembeddings. The Django MongoDB backend will be used to define the models and ingest documents, and MongoDB will be used for lookups, generating and syncing embeddings automatically, and the LLM will return documents based on the retrieved results.

By the end of this tutorial, you’ll build the retrieval stage of a RAG pipeline: a semantic search over Federal Register documents where auto-embeddings handle the embedding work for you. The documents are embedded when you insert them, re-embedded when they change, and your queries are embedded at search time, all without writing embedding code.

There are two ways to follow this tutorial.

Setup

Before starting, make sure you have:

Once these prerequisites are complete, the next step is to create your project folder.

Create a project and virtual environment

Create an empty folder for your project and cd into it.

mkdir regulations_qa
cd regulations_qa

This is your project’s root directory. Create and activate your virtual environment here, if you haven’t already:

python -m venv .venv

To activate the virtual environment on macOS or Linux, run:

source .venv/bin/activate

On Windows, run:

.venv\Scripts\activate

Install Dependencies

Create a requirements.txt file at the root of the project with the following contents:

# requirements.txt
django-mongodb-backend
python-dotenv
requests #to fetch documents from the Federal Register API.

Then install everything:

pip install -r requirements.txt

Create Django Project and Django App

From your project's root directory, run the following command to create a new Django project and app. Add the trailing . in the startproject command, which tells Django to install the project into the current directory instead of creating a new nested folder.

django-admin startproject regulations_qa --template https://github.com/mongodb-labs/django-mongodb-project/archive/refs/heads/6.0.x.zip .

Create the documents app

python manage.py startapp documents

Register the app in regulations_qa/settings.py by adding it to INSTALLED_APPS:

INSTALLED_APPS = [
    'regulations_qa.apps.MongoAdminConfig',
    'regulations_qa.apps.MongoAuthConfig',
    'regulations_qa.apps.MongoContentTypesConfig',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_mongodb_backend',
    'documents',
]

Configure environment variables

Create a .env file at the root of the project with your own values, and list it in .gitignore so it isn’t committed when you push.

# .env
SECRET_KEY=[REPLACE_WITH_YOUR_SECRET_KEY]
MONGODB_URI="mongodb+srv://<username>:<password>@<cluster>.mongodb.net/?retryWrites=true&w=majority"
DB_NAME=regulations_qa
VECTOR_INDEX_NAME=autoembed_index

Delete the original hardcoded SECRET_KEY = '...' line from settings.py, copy the value Django generated in settings.py when you run startproject and replace it in your .env file. It’s hardcoded there by default, and we move it into .env to keep it out of version control, since Django uses it to sign session cookies, CSRF tokens, and other security-sensitive data.

Open regulations_qa/settings.py and make sure it contains all of the following. The import lines and load_dotenv() go at the top. Replace the existing SECRET_KEY and DATABASES definitions, and add the VECTOR_INDEX_NAME line:

import os
from dotenv import load_dotenv

load_dotenv()

SECRET_KEY = os.getenv("SECRET_KEY")
VECTOR_INDEX_NAME = os.getenv("VECTOR_INDEX_NAME")

DATABASES = {
    'default': {
        'ENGINE': 'django_mongodb_backend',
        'HOST': os.getenv("MONGODB_URI"),
        'NAME': os.getenv("DB_NAME"),
    },
}

Create folders for management commands

Next, we create folders with empty __init__.py files for management commands, because Django only discovers custom commands in a management/commands/ package.

mkdir -p documents/management/commands documents/services && touch documents/management/__init__.py documents/management/commands/__init__.py documents/services/__init__.py

The source of data: The Federal Register documents

In this tutorial, we will index documents from the Federal Register; the US government’s daily publications of rules, proposed rules, and notices. The source of data is a public API consisting of documents that have a title and a short prose abstract, which works well with semantic search.

API documentation: https://www.federalregister.gov/developers/documentation/api/v1

The records have fields such as title, type, abstract, document_number (a unique ID), publication_date, html_url, and an agencies list.

  • Some documents contain null values, so we will omit them.
  • Abstracts contain markup like <INF>2.5</INF> in “PM2.5”, so we strip tags before saving.
  • Results are paged at up to 100 per request, and each page includes a next_page_url (cursor-based) to follow.

Test the endpoint via shell

curl -g "https://www.federalregister.gov/api/v1/documents.json?conditions[agencies][]=environmental-protection-agency&per_page=2"

NB: This is public US government data used for demonstration. It is not a legal source; for legal research, use the official edition.

Ingest the Data

We will start by creating models in the models.py file.

# documents/models.py
from django.db import models


class Document(models.Model):
    document_number = models.CharField(max_length=32, unique=True)
    title = models.TextField()
    doc_type = models.CharField(max_length=32)
    agency = models.CharField(max_length=128)
    publication_date = models.DateField()
    abstract = models.TextField()
    source_url = models.URLField(max_length=500)

    def __str__(self):
        return f"{self.document_number}: {self.title[:60]}"

Next, we create and run migrations.

python manage.py makemigrations documents
python manage.py migrate

Next, let’s move to the ingest part.

# documents/management/commands/ingest_federal_register.py

import re
import time

import requests
from django.core.management.base import BaseCommand

from documents.models import Document

API_URL = "https://www.federalregister.gov/api/v1/documents.json"
TAG_PATTERN = re.compile(r"<[^>]+>")


class Command(BaseCommand):
    help = "Load Federal Register documents into MongoDB."

    def add_arguments(self, parser):
        parser.add_argument("--agency", default="environmental-protection-agency")
        parser.add_argument("--limit", type=int, default=300)

    def handle(self, *args, **options):
        params = [
            ("conditions[agencies][]", options["agency"]),
            ("conditions[type][]", "RULE"),
            ("conditions[type][]", "PRORULE"),
            ("per_page", "100"),
        ]
        headers = {"User-Agent": "regulations-qa-tutorial"}
        url, processed = API_URL, 0

        while url and processed < options["limit"]:
            response = requests.get(
                url,
                params=params if url == API_URL else None,
                headers=headers,
                timeout=30,
            )
            response.raise_for_status()
            page = response.json()

            for item in page["results"]:
                abstract = TAG_PATTERN.sub("", item.get("abstract") or "").strip()
                if not abstract:
                    continue
                agencies = item.get("agencies") or [{}]
                Document.objects.update_or_create(
                    document_number=item["document_number"],
                    defaults={
                        "title": TAG_PATTERN.sub("", item["title"]),
                        "doc_type": item["type"],
                        "agency": agencies[0].get("name", ""),
                        "publication_date": item["publication_date"],
                        "abstract": abstract,
                        "source_url": item["html_url"],
                    },
                )
                processed += 1
                if processed >= options["limit"]:
                    break
            url = page.get("next_page_url")
            time.sleep(1)

        self.stdout.write(f"Documents in database: {Document.objects.count()}")

Run the command:

python manage.py ingest_federal_register --limit 100

You can confirm the import in the MongoDB Atlas UI:

MongoDB Atlas Data Explorer showing 100 Federal Register documents in the regulations_qa.documents_document collection

Define the Auto-Embed Index

The vector search index tells MongoDB which field to perform embeddings on and specifies what model should be used. After that, MongoDB automatically generates and stores the embeddings.

  1. In the Atlas UI, open your cluster, go to Search & Vector Search, and click Create Search Index.

Search & Vector Search page in MongoDB Atlas with the Create Search Index button

  1. Choose Vector Search as the search type and Automated Embedding as the way to set up your vector data.

Atlas index configuration with Vector Search and Automated Embedding selected

  1. Choose the JSON editor, select your database and the documents_document collection, and name the index autoembed_index (the name must match VECTOR_INDEX_NAME in your .env). Paste this definition:
{
  "fields": [
    {
      "type": "autoEmbed",
      "modality": "text",
      "path": "abstract",
      "model": "voyage-4-lite"
    }
  ]
}

Atlas JSON editor with the autoEmbed index definition for the abstract field using voyage-4-lite

Review the definition, then create the index.

Review step for the autoembed_index vector search index on regulations_qa.documents_document

  1. Wait until the index status shows it’s ready to query, as the embeddings are being generated in the background.

Atlas Search & Vector Search list showing autoembed_index with status READY and queryable

Query in Plain Text

For auto-embeddings, we use the $vectorSearch stage with a plain-text query and embed the question with the same model the index uses, so there’s no embedding call in our code.

To achieve this, we will create a service module.

# documents/services/search_service.py
from functools import lru_cache

from django.conf import settings
from pymongo import MongoClient

from documents.models import Document

DEFAULT_LIMIT = 5


@lru_cache(maxsize=1)  # create the client once and reuse it
def _get_client():
    return MongoClient(settings.DATABASES["default"]["HOST"])


def _get_collection():
    db_name = settings.DATABASES["default"]["NAME"]
    return _get_client()[db_name][Document._meta.db_table]


def _get_index_name():
    index_name = settings.VECTOR_INDEX_NAME
    if not index_name:
        raise ValueError("VECTOR_INDEX_NAME is not set. Add it to your .env file.")
    return index_name


def semantic_search(question: str, limit: int = DEFAULT_LIMIT) -> list:
    """Search with a plain-text question.

    There is no query vector to pass in: MongoDB embeds the question
    with the index's model.
    """
    pipeline = [
        {
            "$vectorSearch": {
                "index": _get_index_name(),
                "path": "abstract",
                "query": question,
                "numCandidates": limit * 20,
                "limit": limit,
            }
        },
        {
            "$project": {
                "_id": 0,
                "document_number": 1,
                "title": 1,
                "abstract": 1,
                "source_url": 1,
                "score": {"$meta": "vectorSearchScore"},
            }
        },
    ]
    return list(_get_collection().aggregate(pipeline))
  • _get_client() creates the connection to MongoDB once and reuses it. lru_cache remembers the result of the first call, so every search shares one client instead of opening a new connection each time.
  • _get_collection() picks the regulations_qa database and the documents_document collection, using the same settings Django uses.
  • _get_index_name() reads the index name from settings and gives a clear error if it’s missing.
  • semantic_search() sends the question as text in the query field. The second stage trims the output to the fields we need and adds the similarity score.

Then we add a helper to search the docs:

# documents/management/commands/search_docs.py
from django.core.management.base import BaseCommand

from documents.services.search_service import semantic_search


class Command(BaseCommand):
    help = "Run a plain-text search."

    def add_arguments(self, parser):
        parser.add_argument("question")

    def handle(self, *args, **options):
        for hit in semantic_search(options["question"]):
            self.stdout.write(
                f"{hit['score']:.3f}  {hit['document_number']}  {hit['title'][:80]}"
            )

Now, to test, run:

python manage.py search_docs "limits on pollution from factories"

With the example, we select questions that don’t share keywords with the abstracts. A question about “limits on pollution from factories” should retrieve documents about emission standards.

NB: On a free (M0) cluster without a payment method, query embedding requests may be rate limited.

Demo: Edit a Document and Watch It Re-Sync

For the demo, we can either use the Search Tester in MongoDB Atlas, which is tied to the autoembed_index we created previously on the regulations_qa.documents_document collection, or the Python shell.

Atlas Search Tester for autoembed_index on regulations_qa.documents_document

Step 1: Search via MongoDB Atlas and the shell

To test this in the MongoDB Atlas UI, enter this query:

which states need to clean up their air

Search Tester results for the query which states need to clean up their air, with similarity scores

To test via the shell:

Open the shell with python manage.py shell, then run:

from documents.models import Document
from documents.services.search_service import semantic_search
Q = "which states need to clean up their air"
hits = semantic_search(Q, limit=10)
print("\n".join(f"{i}  {h['score']:.3f}  {h['document_number']}  {h['title'][:70]}" for i, h in enumerate(hits, 1)))

Django shell output listing the top 10 semantic search results and their scores

The output should be the same as what we saw via the GUI.

Step 2: Edit the document

target = hits[-1]
doc = Document.objects.get(document_number=target["document_number"])
original = doc.abstract
doc.abstract = original + " This action also identifies which states and areas must clean up their air to meet health-based standards."
doc.save()
print(target["document_number"], "edited")

Django shell appending a sentence to the abstract of document 2026-16515 and saving it

Step 3: Search again

Auto-embeddings are generated asynchronously, so wait 30–60 seconds, then run the query again:

hits = semantic_search(Q, limit=10)
print("\n".join(f"{i}  {h['score']:.3f}  {h['document_number']}  {h['title'][:70]}" for i, h in enumerate(hits, 1)))

Search results after the edit, with document 2026-16515 moved from 10th to 5th place

Now, we should expect to see the edited document rank higher than before.

Run the same query in the Search Tester to see the results via the GUI.

What happened was that MongoDB noticed that the indexed field changed and regenerated that document’s embedding, so document 2026-16515, which was the 10th document in the sample, was ranked higher and placed at the 5th position after the edit and sync.

Wrap-up

Congratulations for making it to this point! We have a docs Q&A assistant using data where MongoDB handles the embedding workflow. This was achieved in the following steps:

  • Insert: documents loaded through the Django ORM
  • Index: one index definition, created in the Atlas UI, naming the field and the model.
  • Query: plain-text questions through a small service without adding a separate embedding call.
  • Edit: changes to the indexed field are re-synced without a re-embedding script.

NB: Data in this tutorial comes from the Federal Register, which publishes US government documents. This tutorial is independent and not affiliated with or endorsed by the Federal Register.

References

Afi Gbadago

Author

Afi Gbadago

Afi advocates for developers at MongoDB with a focus on Django and Python, helping engineers build with modern data technologies through tutorials, talks, and community engagement. A recognized Microsoft MVP for Python and the Web, she serves on the Django Software Foundation board and is Programme Chair for DjangoCon US 2025. Afi is also a Regional Executor for Black Python Devs and an active open-source contributor and international speaker.

Topics

Related

Tutorial

How to Implement Semantic Search in MongoDB