Whisper Latency Comparison: GPU vs CPU vs Cloud RTF Benchmarks

Benchmark Notice: Tested across standard consumer and workstation hardware including NVIDIA RTX 4070 (DirectML/CUDA), Apple M3 Max (Metal), and Intel Core Ultra 7 (NPU/APU). All benchmarks measure total wall-clock time from key release to cursor text injection.

The Physics of Dictation Latency

When evaluating voice dictation tools, perceived speed is the single most critical factor determining whether speaking feels faster than typing on a mechanical keyboard. An average typist operates at 60–80 words per minute (WPM), while natural human speech flows at 150–220 WPM.

However, if a dictation tool introduces a 600ms latency pause between when you finish speaking and when characters appear on screen, the cognitive flow is broken.

There are two fundamentally different architectural pipelines for voice transcription:

Cloud Pipeline:
[Microphone] → [PCM Audio Buffer] → [Opus Encoder] → [TLS Packet] → [Public Internet (30-150ms)] → [Cloud Ingress Queue] → [Server GPU Inference (80-200ms)] → [TLS Stream] → [Desktop App] → [Cursor Injection]
Total Latency: 450ms – 1,200ms (High Jitter)

Local-First Pipeline (HushWrite):
[Microphone] → [Ring Buffer RAM] → [Local GPU whisper.cpp (40-120ms)] → [Win32/macOS SendInput] → [Active Cursor]
Total Latency: 120ms – 180ms (Deterministic)

Hardware Latency Matrix (Real-Time Factor Benchmarks)

Real-Time Factor (RTF) measures the ratio of processing time to the duration of the audio clip. An RTF of 0.10x means a 10-second voice dictation takes exactly 1.0 second to transcribe.

Why Network Jitter Destroys Cloud Dictation

Cloud dictation providers frequently advertise raw model inference speeds of "sub-100ms on cloud H100 clusters." However, this metric ignores the physical reality of the internet packet journey:

Quantization: The Sweet Spot Between Memory and Word Error Rate (WER)

Running full-precision FP32 or FP16 Whisper models locally requires significant VRAM (up to 3GB for large-v3). HushWrite utilizes integer quantization (Q4_0, Q5_1, and Q8_0) to achieve near-lossless accuracy with 60% lower memory bandwidth:

Model Comparison (WER vs Memory Footprint):
• Whisper Base (Q5_1):  92 MB VRAM  | 4.8% WER  | 95ms decode
• Whisper Small (Q5_1): 190 MB VRAM | 3.1% WER  | 145ms decode (Default Starter)
• Whisper Large-v3-Turbo (Q5_0): 540 MB VRAM | 1.8% WER | 185ms decode (Pro Tier)
• Full FP16 Large-v3:  3,100 MB VRAM | 1.7% WER | 380ms decode

For 98% of professional writing tasks (emails, pull requests, Slack chats, and documentation), small-q5_1 offers the optimal balance of instantaneous tail response and state-of-the-art vocabulary comprehension.

Conclusion & Reproducibility

Local hardware inference has decisively won the latency race. By leveraging DirectML on Windows and Metal on Apple Silicon, HushWrite delivers sub-150ms tail insertion times that cloud-streaming architectures cannot match due to fundamental networking constraints.