A floor that talks faster than any QA team can listen
JustDial's telemarketing teams across India generate tens of thousands of recorded sales calls a day. Call quality decides renewals — but checking it meant a human QA desk sampling a tiny fraction of calls, days late, with rubrics that drifted between auditors.
- Fairness, not just volume. Sampling naively audits the chattiest agents again and again while others go unreviewed for weeks.
- Every team scores differently. Audit rubrics vary by city, team and product line — one prompt cannot audit them all.
- GPU minutes are contested. Transcription and LLM scoring share GPU infrastructure with other production workloads; capacity spikes cause silent failures.
- The pipeline must explain itself. When counts dip, operations needs to know exactly which stage lost calls, and why, the same day.
Five stages, one honest funnel
The system is a staged pipeline where every call is either audited or accounted for — no silent drops. The stages below run continuously through the working day.
Ingest & filter
Dialer call logs stream into MongoDB. Calls are filtered to genuine sales conversations — active, recorded, and long enough to be worth auditing.
Fair round-robin sampling
One call per agent per cycle, newest first, with longer (5–15 minute) conversations prioritized — so every agent gets audited and GPU time goes to the calls that matter. A backlog gate pauses intake when the transcription queue is full.
GPU transcription
A Whisper ASR pool turns each recording into a speaker-tagged transcript, feeding a queue that downstream stages drain continuously.
LLM audit scoring
An LLM scores each transcript against the audit template mapped to that agent's team, city and product line — the same rubric a human auditor would use, applied identically every time.
Reporting & self-healing
Dashboards reconcile every stage daily; a retry service re-queues failed transcriptions and repairs stale recording paths automatically, with no human in the loop.
What actually got built
Fair sampler service
A round-robin scheduler that guarantees per-agent coverage across thousands of telemarketers, prioritizes 5–15 minute conversations, and self-throttles against the GPU queue. Its hot query was rebuilt around an index-only scan — from over 3 minutes to 3 seconds.
Audit template mapping config
A rules engine maps every call to the right audit rubric by team, city and product line, so the LLM audits each call the way that team's own QA desk would.
Self-healing retries daily job
A reconciliation service that finds every failed transcription of the last 24 hours, repairs recordings whose file format changed on disk, and re-queues them — it recovered 14,000+ calls lost to GPU capacity incidents without manual intervention.
Count reconciliation reporting
Daily funnel reports that account for every single call from dialer log to finished audit — each gap quantified and classified as permanent-by-design or recoverable-in-flight.
Numbers from production
What this system taught me
- Fairness is a scheduling problem, not a sampling problem. "Audit more calls" is easy; "audit every agent regularly" required designing the round-robin scheduler around people, not documents.
- Shared GPUs fail in bursts, not gradually. Memory over-commitment between co-tenant models causes sudden minutes-long failure storms. Design for burst recovery — a daily self-healing pass beat every attempt at perfect prevention.
- Reconcile counts end to end, every day. A funnel where every drop is named ("inactive", "recording unavailable", "queued") turns "why are audits down?" from a day of investigation into a one-look answer.
- Verify what the code does, not what it looks like it does. A dead sort key sat in the sampler looking like a duration preference for months. Only cohort-level data analysis revealed the actual selection behavior.
- Databases reward queries shaped like their indexes. Rewriting one distinct-scan aggregation took the sampler's hot query from 3 minutes of collection scanning to 3 seconds of index-only work — the biggest single win in the system.