Back to Case Studies

On-Prem Clinical Transcription with 99.1% Accuracy on Medical Vocabulary

Executive Summary

A hospital network needed accurate, real-time clinical transcription without sending sensitive physician audio outside its network. GenAI Protos fine-tuned Whisper large-v3 on 800 hours of de-identified clinical audio and deployed the solution on the hospital's on-prem GPU infrastructure. The solution achieved 99.1% word error rate performance on clinical vocabulary, saved an average of 42 minutes of documentation time per physician per day, and required zero outbound audio transfer.

01Client Requirement

The hospital's physicians were spending more than 90 minutes each day on clinical documentation. The client needed: Real-time transcription of physician dictation Higher accuracy for medical terminology, drug names, dosages, and ICD-10 codes Clinical-grade performance beyond generic speech-to-text models Audio and inference to remain within the hospital's network A deployment approach aligned with HIPAA and compliance requirements No recurring per-minute SaaS transcription fees

Section image

02Our Approach

We built a compliance-first, domain-adapted transcription pipeline, starting with 800 hours of physician dictation prepared using Safe Harbor de-identification so that all PHI was removed before model training. We then fine-tuned Whisper large-v3 using LoRA/PEFT and augmented its tokenizer with 14,000 clinical terms covering drugs, conditions, procedures, and abbreviations. To structure the output, we added speaker diarisation to distinguish participants and section detection for SOAP-note components (subjective, objective, assessment, and plan). For live use, we built a TensorRT-optimized streaming pipeline with WebSocket-based delivery, 200 ms audio chunks, and partial-result correction. Finally, the entire inference system was deployed on-premises inside the hospital VPC on 4× A100 GPUs, with zero outbound network access for the inference path.

03Outcome

  • 99.1% accuracy performance on clinical vocabulary
  • Improved from a 92.4% baseline
  • 42 minutes of documentation time saved per physician per day
  • 0 audio leaving the hospital network
  • $0 per-minute SaaS transcription fees
  • On-prem deployment supporting the client's privacy and compliance requirements

Frequently asked questions

Yes. This solution was deployed inside the hospital's VPC with zero outbound network access for the inference path.

Generic speech-to-text models can struggle with specialized terminology. Fine-tuning Whisper on domain-specific clinical data can improve recognition of medical vocabulary, drug names, procedures, and abbreviations.

The model was fine-tuned using 800 hours of de-identified physician dictation aligned with corrected transcripts.

The solution used Whisper large-v3, fine-tuned using LoRA/PEFT with an expanded medical vocabulary.

Have a similar AI, speech-to-text, or private deployment challenge?

Let's scope your use case and determine the right technical approach.

Talk to our AI engineers