Fine-Tuning XTTS v2 for More Natural Bible Narration

The strongest checkpoint in this XTTS v2 fine-tune occurred at global step 10,725. Training continued to step 35,750, but evaluation loss increased and the later checkpoint produced weaker spectral and pitch measurements.

This article explains how I selected the model. It separates training signals from listening observations and documents the limits of the local evaluation.

Objective and baseline

Alveum created an internal challenge to develop an in-house voice for King James Version Bible narration. The target was the natural pacing and phrase-level melody of the existing ElevenLabs voice.

The supplied XTTS v2 baseline used 324 processed Genesis clips, or about 50 minutes of speech. It pronounced the text clearly, but its rhythm and pitch movement sounded robotic.

My main target was prosody: the timing, stress, and pitch pattern that gives a sentence its spoken shape. I evaluated intelligibility, pacing, pitch behavior, spectral similarity, and reproducibility.

Model and dataset decisions

I first tested Fish S2 because a newer architecture could have improved naturalness. In my listening tests, its early output did not exceed the 324-clip XTTS baseline, despite using more compute.

I stopped that experiment and returned to XTTS v2. Because I did not evaluate Fish S2 and XTTS with the same controlled pipeline, I treat this as an architecture decision rather than a quantitative benchmark.

XTTS also matched the project constraints. The baseline already provided compatible training code, inference code, dataset conventions, and a portable checkpoint format.

I expanded the processed dataset from 324 Genesis clips to 3,611 Bible and Gospel clips. The trainer used 3,575 clips for training and held out 36 clips for evaluation.

The trainer configuration and log confirm these counts. Dataset size alone does not measure quality: transcript accuracy and clip boundaries also affect pronunciation, timing, and rhythm.

The run fine-tuned the XTTS v2 GPT component. The training log reports 441,018,563 parameters and one GPU.

Training behavior and checkpoint selection

At step 1,000, the logged sample loss was 2.4582, down from 4.0892 at step 0. These are individual training samples, not a stable evaluation series, so they only show that optimization was active.

At about the same stage, I heard less-flat pitch movement and better continuity between phrases. That is a listening observation; the training loss does not prove that prosody improved.

The held-out evaluation set provided a stronger checkpoint-selection signal. The Gospels run reached its minimum evaluation loss of 2.8185 at step 10,725, then worsened in every remaining epoch.

At the final step, 35,750, evaluation loss had increased to 3.0925. That value is 9.7% higher than the minimum, even though training had continued for seven more epochs.

The same pattern appeared in the smaller Genesis run. Its minimum occurred at step 5,652, before later checkpoints degraded.

I used best_model_10725.pth as the submitted model. I also compared four John 3:16 samples: the base model, an early fine-tune, the selected model, and checkpoint 34,000.

Listen for intelligibility, pause placement, pitch movement, and stability across the same sentence. Use headphones for the clearest comparison.

Evaluation and inference tuning

I used duration, an approximate Mel Cepstral Distortion value, and F0 deviation to screen candidates. I then used listening tests for the final selection.

MCD estimates distance between spectral envelopes. F0 deviation compares pitch tracks in cents, where 100 cents equals one semitone. Lower values mean closer matches within this evaluation pipeline.

The MCD calculation was local and approximate. Its absolute values are not comparable with published benchmarks; only relative changes from the same script, reference, and test sentence are useful here.

On Genesis 1:1, checkpoint 34,000 was closer to the reference duration. The validation-best model had a 20.6% lower MCD value and a 7.0% lower F0 deviation.

Inference settings also changed the output. On Exodus 2:2, increasing speed from 1.00 to 1.25 reduced the duration error from 3.065 seconds to 0.337 seconds.

I then swept temperature and top_p at speed 1.25. Temperature controls sampling randomness, while top_p limits sampling to the most probable token set whose cumulative probability reaches the threshold.

The 0.85 / 0.92 setting produced the widest pitch range, but 0.95 / 0.95 had the lowest MCD and F0 deviation. I selected the latter as the better balance of similarity, pacing, and intelligibility.

The final inference defaults were:

speed = 1.25
temperature = 0.95
top_p = 0.95

Packaging and reproducibility

The selected checkpoint was approximately 5.6 GB, so I kept it outside Git. A reviewer places the following files in model/gospels_best/:

model.pth
config.json
vocab.json

The reviewer package includes a Python 3.11 environment, pinned dependencies, a local copy of TTS 0.22.0, CLI inference, a web interface, Docker configuration, and Windows and macOS instructions.

I tested the package in a new virtual environment with the same command provided to reviewers:

python scripts/inference.py \
  --text "In the beginning God created the heaven and the earth." \
  --reference reference_audio/reference.wav \
  --checkpoint model/gospels_best \
  --output output/reviewer_test.wav \
  --speed 1.25 \
  --temperature 0.95 \
  --top-p 0.95

The smoke test produced a 4.821-second, mono, 24 kHz PCM-16 WAV file. This verified that the model could load and generate audio outside the training environment.

Results and limitations

The selected XTTS model improved on the starter baseline in my listening tests and remained the strongest candidate by validation behavior and local checkpoint comparisons.

It did not reach the phrase-level melody of the ElevenLabs target. Punctuation improved pause placement but could not fully control pitch shape, and output quality remained sensitive to the reference clip.

The evaluation also has limits: only 36 clips were held out, the MCD implementation was approximate, no formal listening panel was used, and the official hidden test sentences were unavailable during packaging.

The next evaluation should use a larger held-out set and a blinded A/B listening test across short, medium, and long passages.

The submission later received first place in the Alveum AI Voice Model Challenge. More importantly, the evidence showed why the best result came from an earlier checkpoint and a tuned inference pipeline, not from more training alone.