I was debugging a client-side transcription feature when the model kept mis-hearing a single speaker from Lagos. The same audio file, fed to Whisper on my laptop, produced a perfect transcript. On the production server, the output was riddled with "uh" insertions and completely wrong nouns. After a few false starts, I realized the problem wasn't the server hardware, it was the way I was measuring Whisper's performance on a regional accent.

What follows is the exact workflow I now use whenever I need to know whether Whisper (or any other open-source model) is biased against a particular dialect. The steps are concrete, the code runs on a vanilla Python 3.10 install, and the pitfalls are highlighted so you don't repeat my afternoon-long debugging session.

Why bias matters in speech-to-text

Transcription is the backbone of many products: captioning videos, voice-controlled forms, and accessibility tools. If a model consistently drops words or substitutes them with unrelated terms for a specific accent, the user experience collapses and legal compliance can be jeopardised. Bias isn't just a moral issue; it's a measurable loss of signal-to-noise ratio that can be quantified with a few simple metrics.

Collecting a representative audio set

The first mistake I made was pulling ten random YouTube clips and calling that a dataset. The sample was too small, too noisy, and not labelled with speaker metadata. Here’s what I do now:

  1. Define the target accent, e.g., Nigerian English, Yoruba-influenced speech.
  2. Source audio from open repositories, Common Voice, VoxPopuli, or community-contributed podcasts.
  3. Filter by duration and quality, keep clips between 5 and 30 seconds, 16 kHz mono, <-3 dB RMS.
  4. Manually verify transcripts, a quick sanity check in a spreadsheet ensures the ground truth is trustworthy.

A typical set ends up with 50-100 clips, enough to smooth out random variation but still light enough to run on a single GPU.

Setting up Whisper for reproducible runs

Whisper ships as a pip package, but the default whisper.transcribe function adds random padding to the audio, which can shift word boundaries. To freeze that behaviour, I call the model directly through the decode API.

# whisper_bias_test.py
import whisper, torch, json, pathlib

MODEL_NAME = "base"
model = whisper.load_model(MODEL_NAME).to("cuda")

def transcribe(audio_path: pathlib.Path) -> str:
    # Load and pad manually, no random jitter
    audio = whisper.load_audio(str(audio_path))
    mel = whisper.log_mel_spectrogram(audio).to(model.device)
    # Force greedy decoding for deterministic output
    options = whisper.DecodingOptions(language="en", without_timestamps=True, fp16=False)
    result = whisper.decode(model, mel, options)
    return result.text.strip()

if __name__ == "__main__":
    data_dir = pathlib.Path("./audio_samples")
    out = {}
    for wav in data_dir.glob("*.wav"):
        out[wav.name] = transcribe(wav)
    pathlib.Path("results.json").write_text(json.dumps(out, indent=2))
  • whisper.load_audio reads the file at 16 kHz, matching the model's expectations.
  • whisper.log_mel_spectrogram produces the exact same mel frames Whisper would see internally.
  • DecodingOptions disables beam search and timestamps, removing sources of nondeterminism.
  • The script writes a JSON map of file names to transcriptions for later comparison.

Computing error metrics that surface bias

Word Error Rate (WER) is the industry standard, but it can mask systematic errors when the reference transcript is short. I combine three metrics:

  1. WER, overall accuracy.
  2. Insertion rate of filler words, Whisper often adds "uh" or "um" for accented speech.
  3. Semantic similarity, using sentence-transformers to catch cases where the wording changes but the meaning stays.
# evaluate_bias.py
import json, re
from jiwer import wer
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

def filler_rate(transcript):
    fillers = re.findall(r"\b(uh|um)\b", transcript.lower())
    return len(fillers) / max(len(transcript.split()), 1)

def semantic_score(ref, hyp):
    emb_ref = model.encode(ref, convert_to_tensor=True)
    emb_hyp = model.encode(hyp, convert_to_tensor=True)
    return util.pytorch_cos_sim(emb_ref, emb_hyp).item()

with open('results.json') as f:
    preds = json.load(f)

# Load ground-truth from a parallel JSON file
with open('ground_truth.json') as f:
    refs = json.load(f)

stats = []
for fname, hyp in preds.items():
    ref = refs[fname]
    stats.append({
        "file": fname,
        "wer": wer(ref, hyp),
        "filler_rate": filler_rate(hyp),
        "semantic": semantic_score(ref, hyp)
    })

print(json.dumps(stats, indent=2))
  • wer gives the raw edit distance.
  • filler_rate isolates the symptom I first saw on the production server.
  • semantic_score flags transcripts that look different but still convey the same idea, useful for downstream tasks that only need intent.

Interpreting the numbers

When I ran the script on a 60-clip Nigerian set, the average WER was 18 %, compared to 9 % on a US-English control set of the same size. More tellingly, the filler rate jumped from 0.2 % to 7 %. Semantic similarity stayed above 0.85 for both groups, meaning Whisper was still capturing the gist but adding noise that would break caption timing or voice-assistant commands.

These three numbers together tell a story: Whisper understands the content but struggles with the phonetic patterns of the accent, inserting spurious tokens that degrade downstream pipelines.

Mitigation strategies you can apply today

  1. Fine-tune on the target accent, Whisper's base models accept a small amount of supervised data. Even a few hundred minutes of clean audio can halve the filler rate.
  2. Post-process with a filler filter, a simple regex that strips "uh"/"um" after decoding, followed by a language-model re-scoring step to restore naturalness.
  3. Hybrid inference, run Whisper for the bulk of the transcript, then pass only the segments with high filler density to a smaller, accent-specific model like wav2vec-2.0 fine-tuned on the same data.
  4. User-feedback loop, expose a correction UI and feed the corrected text back into a periodic re-training job.

The cheapest win is the filler filter; it removes the most visible symptom with virtually no compute cost. Fine-tuning gives the biggest long-term accuracy boost but requires a GPU and a labelled dataset.

Common pitfalls and how to avoid them

  • Using the wrong language code, Whisper defaults to multilingual detection; forcing language="en" prevents it from swapping to a different script mid-stream.
  • Mixing sample rates, feeding a 44.1 kHz file without resampling leads to stretched audio and inflated WER.
  • Relying on a single metric, a low WER can hide high filler rates, which matter for real-time applications.
  • Over-fitting during fine-tuning, keep a validation split that mirrors the production accent distribution; otherwise you may improve on your test set but regress on live traffic.

Scaling the test to CI pipelines

Once the script works locally, I add it to the project's GitHub Actions workflow. The job runs on a GPU-enabled runner, pulls the latest audio fixtures, and fails the build if the filler rate exceeds a threshold (e.g., 3 %). This early warning prevents regressions when Whisper is upgraded or when the preprocessing pipeline changes.

# .github/workflows/whisper-bias.yml
name: Whisper Bias Check
on: [push, pull_request]
jobs:
  bias-test:
    runs-on: ubuntu-latest
    container:
      image: nvidia/cuda:12.1.0-runtime-ubuntu22.04
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - run: pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
      - run: pip install -r requirements.txt
      - run: python whisper_bias_test.py
      - run: python evaluate_bias.py | tee stats.json
      - name: Fail on high filler rate
        run: |
          python -c "import json, sys; data=json.load(open('stats.json')); 
          max_rate=max(item['filler_rate'] for item in data); 
          sys.exit(1) if max_rate>0.03 else sys.exit(0)"
  • The workflow spins up a CUDA container so the model runs at speed.
  • The final step parses the JSON output and aborts the CI if the filler threshold is crossed.

FAQ

How many audio samples do I need for a reliable bias measurement?

A few dozen well-balanced clips (5-30 seconds each) are enough to surface systematic filler insertion. For tighter confidence intervals, aim for 100+ samples spread across speakers and recording conditions.

Does Whisper's multilingual mode help with accent bias?

Sometimes, but it adds a language-identification step that can misclassify accented English as another language, increasing errors. Explicitly setting language="en" is usually safer when you know the spoken language.

Can I use the same script for other models like Azure Speech or Google Cloud?

Yes, replace the transcribe function with the respective SDK call, keep the same audio preprocessing, and the evaluation block will work unchanged.

Wrapping up

If you ever see a flood of "uh" in your captions, the first thing to check is whether the model is being evaluated on the right accent. A reproducible test suite, a few concrete metrics, and a simple CI gate keep the problem visible before it reaches users. Once you have the numbers, you can decide whether a quick post-process filter is enough or whether a fine-tuned model is worth the investment. Either way, you’ll have moved from "it works for me" to "it works for everyone".