Speaker diarization Splitting a recording into turns by voice, without knowing anyone’s name.
Short answer. Speaker diarization is the task of working out who spoke when in a recording with several voices: it marks where each person’s turns start and end and gives each speaker an anonymous label such as speaker 0 or speaker 1. It does not name anyone, and it is separate from transcription. A typical pipeline finds the speech, splits it where the voice changes, and groups the pieces by voice. The open-source pyannote toolkit reports diarization error rates of 17.0% on AMI meeting recordings with close microphones and 19.9% with a single distant microphone, so expect mistakes where people talk over each other.
01
How it works
Jurafsky and Martin describe diarization systems that use voice activity detection to find stretches of continuous speech, extract a speaker embedding for each stretch, and cluster the embeddings so that segments likely to come from one speaker are grouped. pyannote’s documentation lists the same steps: speech activity detection, speaker change detection, clustering, and an optional step that identifies each cluster. The output is a list of start and end times with labels such as speaker_0.
Naming the speakers is a further step. The textbook separates speaker verification (is this person X?) from speaker identification (which of many known people is this?). Both need earlier voice samples, and a person can also just rename the labels after listening.
02
How error is measured
Diarization error rate (DER) is the standard metric. pyannote’s documentation defines it as false alarm plus missed detection plus speaker confusion, divided by the total reference speech, so a DER of 17% means that roughly 10 minutes of every 60 minutes of speech are missed, wrongly added or given to the wrong person. Researchers often ignore a 500 ms collar around each turn boundary, because human labels cannot be exact. DER counts overlapping speech, so a system that cannot detect overlap loses points there.
Benchmark (pyannote community-1)
Diarization error rate
VoxConverse
11.2%
AMI, individual headset microphones
17.0%
AMI, single distant microphone
19.9%
DIHARD 3
20.2%
AVA-AVD
44.6%
03
Why it matters for minutes
An action item such as “I will send the revised draft by Friday” is only useful if it is attributed to the right person. If speaker labels are wrong, the owner is wrong. The AMI results above also show the cost of a distant microphone: the same recordings score worse when captured by a single microphone than by headsets.
pyannote reports a speed of 31 seconds of processing per hour of audio for its community-1 pipeline on an NVIDIA H100 data-centre GPU, a much larger card than an office box would use.
04
What to check
Audio: one microphone per person beats one in the middle of the table.
Overlap: ask how the tool handles two people talking at once.
Number of speakers: pyannote lets you give the number, or a minimum and maximum, when you know it.
Privacy: pyannote’s pipelines run locally once the model has been downloaded, which needs you to accept the model’s conditions on Hugging Face. The toolkit has optional telemetry, which can be switched off with the PYANNOTE_METRICS_ENABLED environment variable.
Treat a speaker embedding as derived from someone’s voice, with the same care as the recording.
Whisper’s model card says it has not been robustly evaluated for speaker diarization, so do not expect it from a transcription model alone.
05
Where ANVE stands
ANVE’s minutes maker needs the owner of each action item to be confirmed by a person before the item is approved, and each confirmation is logged. The question of who said what is therefore settled by a person, not guessed by a model. Every action item also has to quote the transcript line it came from.
06
Questions people ask
Is diarization the same as speaker recognition?
No. Diarization marks where different speakers talk, with anonymous labels. Speaker recognition identifies a person: verification checks a claimed identity, identification picks one person from many.
Does Whisper do diarization?
No. Whisper’s model card says it may show some ability on speaker diarization if fine-tuned, but it has not been robustly evaluated for it. Projects such as WhisperX add diarization on top.
How many speakers can it handle?
There is no fixed limit, but accuracy falls as the number of speakers and the overlap grow. pyannote lets you set the number of speakers, or a minimum and maximum, when you know them.
Can diarization run offline?
Yes. pyannote’s pipelines run locally once the model is downloaded, which needs accepting the model’s conditions on Hugging Face and a token. Optional telemetry can be disabled.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
