NVIDIA PersonaPlex Tutorial: Run a Natural, Real-Time Local Voice Assistant

Setting Up the Environment 

Before we start, visit the Hugging Face model page for nvidia/personaplex-7b-v1 and accept the model usage conditions. 

PersonaPlex is a gated model, so you will need a Hugging Face API token. Generate the token from your Hugging Face account and keep it handy, as we will add it to the environment variables later to allow model access.

Next, go to RunPod and launch a new A40 GPU pod. Select the latest PyTorch image, then click Edit to customize the environment. 

Increase the container disk size to 50 GB, since the model itself is around 20 GB, and additional dependencies will also be downloaded. In the exposed HTTP ports section, add port 8998. Under environment variables, add HF_TOKEN and paste your Hugging Face API token.

Once everything is configured, save the overrides and deploy the pod.

After the pod starts, you will see a link to the JupyterLab instance. Open it, then launch a terminal. You can access the machine via SSH or the web terminal, but using the Jupyter terminal is the simplest option.

In the terminal, install the Opus audio codec development library, which is required for audio processing:

sudo apt update
sudo apt install libopus-dev

Installing PersonaPlex From Source

The PersonaPlex codebase is open-sourced by NVIDIA and available on GitHub (NVIDIA/personaplex: PersonaPlex code), making it easy to explore, customize, and run locally. 

Installing from source gives you full control over the setup and ensures compatibility with the latest updates in the repository.

To get started, clone the PersonaPlex repository:

git clone https://github.com/NVIDIA/personaplex.git
cd personaplex

Then install Moshi, the core full-duplex speech model that PersonaPlex is built on. 

Moshi is responsible for real-time listening and speaking, enabling PersonaPlex to handle interruptions, pauses, and natural conversational timing without relying on a traditional ASR → LLM → TTS pipeline.

Installing it from source ensures that all audio, streaming, and conversational components are correctly set up for local execution.

pip install moshi/.

Once Moshi is installed, your environment is fully prepared to start the PersonaPlex server and begin interacting with the model in real time.

Starting the PersonaPlex WebUI Server

With the environment and dependencies in place, you can now start the PersonaPlex WebUI server. In the terminal, run the following command to launch the Moshi server that powers PersonaPlex:

python -m moshi.server --host 0.0.0.0 --port 8998

The first time you run this command, the server will automatically download the PersonaPlex model and other required files. This step can take a few minutes, depending on your network speed, as the model is fairly large.

Once the download is complete, the server will start listening on port 8998. 

To access the WebUI, go back to your RunPod dashboard. 

In the Connect section, find the exposed port 8998 and click the link provided. This will open the PersonaPlex WebUI in your browser, where you can start interacting with the model through real-time voice conversations.

Testing PersonaPlex Using the WebUI

The PersonaPlex WebUI comes with several example prompts to help you get started, and you can also create your own custom prompt to define the assistant’s role and behavior. 

You can select a voice before connecting, which determines how the persona sounds during the conversation. 

For this tutorial, it is best to start with the default settings to get a feel for the system.

After clicking the Connect button, you will be taken to the interactive session. Here, you can speak directly to the model, hear its responses in real time, and see a live transcript of the conversation as it happens. 

The experience feels fluid, with natural timing and the ability to interrupt or respond quickly.

During my testing, the interaction felt surprisingly natural and engaging. 

I did notice some occasional stuttering, which was likely due to browser or system load rather than the model itself. Aside from that, the conversation felt very close to speaking with a real person.

Once you are comfortable, try experimenting with different prompts and voices. PersonaPlex supports a wide range of pre-packaged voice embeddings, including more natural conversational voices and more expressive variants:

• Natural (female): NATF0, NATF1, NATF2, NATF3
• Natural (male): NATM0, NATM1, NATM2, NATM3
• Variety (female): VARF0, VARF1, VARF2, VARF3, VARF4
• Variety (male): VARM0, VARM1, VARM2, VARM3, VARM4

Switching voices and prompts is a great way to explore how PersonaPlex maintains personality, tone, and conversational flow across different roles and speaking styles.

Running PersonaPlex Offline With Python

The WebUI is great for real-time conversations, but you can also run PersonaPlex offline from Python. This is useful when you want repeatable outputs, want to test different prompts quickly, or want to generate audio files you can reuse in demos and experiments. 

The offline flow is simple: you provide (1) a voice prompt, (2) a text prompt, and (3) an input WAV file. PersonaPlex then generates an output audio response and a JSON file containing the model’s transcript.

Start a new notebook in Jupyter and change the directory to the cloned repository:

%cd personaplex

Run the following in a new cell. This calls the offline runner and writes both the generated audio and the transcript to disk:

%%capture
!python -m moshi.offline \
  --voice-prompt "NATF2.pt" \
  --text-prompt "You are a wise and friendly teacher. Answer questions in a clear, engaging way." \
  --input-wav "assets/test/input_assistant.wav" \
  --seed 42424242 \
  --output-wav "out_teacher.wav" \
  --output-text "out_teacher.json"

What each flag does:

• --voice-prompt selects the voice embedding (here, a natural female voice).
• --text-prompt defines the role and behavior of the assistant.
• --input-wav is the recorded user audio the model will respond to.
• --seed makes the output more reproducible between runs.
• --output-wav is the generated spoken response.
• --output-text saves the transcript output as JSON.

Once the command finishes, you can play the response audio directly:

from IPython.display import Audio

Audio("out_teacher.wav")

You should hear a clear, natural response in the selected voice, matching the persona defined in your text prompt.

The JSON transcript can sometimes contain tokenization artifacts such as extra spaces around punctuation or split word pieces. The helper below loads the JSON and cleans it into readable text.

import json
import re

with open("out_teacher.json") as f:
    data = json.load(f)

def detokenize(tokens):
    # 1) drop padding-like tokens
    drop = {"PAD", "EPAD", "<pad>", "</s>", "<s>"}
    toks = [t for t in tokens if t not in drop]

    # 2) join with spaces first
    s = " ".join(toks)

    # 3) fix spacing around punctuation
    s = re.sub(r"\s+([.,!?;:])", r"\1", s)

    # 4) fix common split contractions: "it ' s" -> "it's"
    s = s.replace(" ' s", "'s").replace(" n't", "n't").replace(" 're", "'re").replace(" 'm", "'m").replace(" 've", "'ve").replace(" 'd", "'d")

    # 5) fix stray spaces around apostrophes
    s = re.sub(r"\s+'\s+", "'", s)

    # 6) fix cases like "for k" -> "fork" and "fl uff" -> "fluff"
    # (general rule: merge single-letter fragments if they look like split wordpieces)
    s = re.sub(r"\b([A-Za-z])\s+([A-Za-z]{1,3})\b", r"\1\2", s)

    # 7) collapse multiple spaces
    s = re.sub(r"\s{2,}", " ", s).strip()

    return s

clean_text = detokenize(data)  # replace with your list variable
clean_text

You should now see a readable transcript string that matches the generated audio. If your output still contains odd splits (for example, “fl uff” or “afor k”), that is normal for some runs and can be cleaned further with additional rules, but the main content should already be understandable.

"Hey, let me know if you have any questions. 
Hmm, first rinse the rice a couple of times until the water runs clear, that cuts down on starch, then use apot with a tight fitting lid, bring to a boil, 
give it a quick stir, then turn the heat down low and cover, let it s immer without lifting it, and when it'done fl uff it with afor k, that usually 
You could to ss the hot rice with a nice handful of chopped fresh herbs like basil or par sley, or you could sprinkle a little g rated cheese, 
a squeeze of lemon or lime, adr izzle of olive oil, some chopped fresh herbs, or even some to ast ed nuts, that adds color and flavor."

Final Thoughts

Testing PersonaPlex genuinely surprised me. From the very first interaction, it felt less like experimenting with a model and more like having a real conversation. Being able to interrupt naturally, get instant responses, and maintain a consistent personality throughout made it feel far ahead of most voice systems I have tried. Running everything locally made the experience even more impressive, with no noticeable delay or loss of control.

There are still some drawbacks. I noticed occasional stuttering, and once the conversation moves on, it does not always handle returning to an earlier topic smoothly. 

It also does not fully understand non-English accents yet, which can lead to mispronounced names or imperfect transcriptions. These feel like edge cases rather than fundamental issues, and I am confident they will improve quickly.

I am sure that in the future these issues will be resolved, and we will get an even better, fully local and real-time conversational AI that combines the reasoning of GPT-5.2 with the voice quality of ElevenLabs.