What is speech recognition?

Speech recognition, also called automatic speech recognition (ASR), is technology that lets a computer listen to someone speaking and turn their words into text. Dictation on your phone, meeting transcripts, video captions and automated phone menus all rely on it.

It’s genuinely useful, but it doesn’t always hear things correctly. Knowing when it tends to slip helps you record in a way that gets better results, and tells you which parts of a transcript a person should check.

What does speech recognition actually do?

Speech recognition works out which words were spoken and writes them down. Say a piano teacher, walking between lessons, tells their phone: “Next week’s trial lesson is open on Tuesday from four.” The words appear as text, ready to use as the draft of a reply.

Its job ends once the words are on the page. Understanding what the text means, deciding how to answer or summarizing the main points is the work of other systems, such as a large language model (LLM). Most AI you can talk to by voice chains three steps together: speech recognition turns your voice into text, an LLM writes a reply, and text-to-speech reads the reply out loud.

You’ll find speech recognition in situations like these:

• Adding captions to the videos in an online course
• Turning a small company’s meeting into text as a first draft of the minutes
• Dictating a note or message when your hands are busy
• Working out what a caller needs and routing the call to the right person

How does speech recognition work?

Turning sound into numbers

What a microphone picks up is a wave: a record of how the air vibrated. Speech recognition first cuts this wave into very short slices, a few hundredths of a second long, and for each slice writes down how strong the sound is at each pitch. The result is a long series of numbers. A common version adjusts the pitch scale to match how human hearing works, and is called a mel spectrogram.

Picking the most likely words

Next, a model built from a neural network (a system that learns from huge numbers of examples) reads those numbers and picks the most likely sequence of words. Older systems split this work between separate parts: one for sounds, a pronunciation dictionary, and one for how words fit together. Today it’s common for a single model to handle everything from sound to text, after learning from large amounts of audio paired with transcripts.

The surrounding words help a great deal. On its own, the sound of “their” could also be “there” or “they’re”. In “Is their class full?”, the context makes the right spelling clear. Names and unusual words are harder because the context gives fewer clues.

Streaming or batch?

Speech recognition is used in two main ways:

	Streaming (while you speak)	Batch (after recording)
When text appears	Almost as you speak	All at once, after the recording is sent
Good for	Dictation, live captions, voice assistants	Recorded meetings, interviews, videos
What to watch for	Early results can change as more audio arrives	You wait for the result

If you’ve ever seen dictated words change a moment after they appear, that’s streaming at work: the early guess is revised once the system hears what comes next.

Why does accuracy vary?

The same speech recognition system can give very different results depending on how the audio was recorded and how people spoke.

Cause	What tends to happen	What helps
Background sound	Music or other voices are mistaken for words	Speak somewhere quiet
Distance from the microphone	Quiet speech, so words drop out	Keep the microphone close to your mouth
Phone lines and heavy compression	Less detail in the sound, so similar sounds get confused	Record at the original quality where you can
Accents and speaking style	Ways of speaking that were rare in training cause errors	Speak clearly and a little slower
Jargon and names	Swapped for more common words that sound alike	Use a feature that lets you add your own terms
People talking over each other	Overlapping parts go missing or get mixed up	Take turns

One surprise: strong noise reduction applied while recording can make recognition worse. Google Cloud’s best practices for speech data say that noise reduction applied before audio is sent generally lowers accuracy, and recommend placing the microphone as close to the speaker as possible instead.

For jargon and names, many services let you supply words in advance so they’re more likely to be chosen. Adding your class names, course titles and product names can cut down on mix-ups.

How is accuracy measured?

Accuracy is measured by comparing the output with a transcript a person wrote. You count the words that were added (insertions), left out (deletions) and replaced with a different word (substitutions), then divide by the number of words in the correct transcript. This is the word error rate, or WER. For languages written without spaces between words, such as Japanese, a character error rate (CER) is often used instead. Lower is better for both.

The type of error also hints at the cause. Microsoft’s guide to testing speech recognition accuracy notes that many deletions usually point to a weak audio signal, insertions to noise or crosstalk, and substitutions to domain-specific terms the model hasn’t seen enough of.

How are speakers and punctuation handled?

When you transcribe a meeting, you usually want to know who said what. Separating a recording by speaker is called speaker diarization. The system notices where the voice changes, based on differences in how voices sound, and labels each one with a number, such as “Speaker 1” and “Speaker 2”. It doesn’t know who those people are, so a person adds the names. Voices that sound alike, and moments when people talk over each other, are where it most often gets confused.

Punctuation isn’t part of the sound itself. Speech recognition that adds punctuation guesses where periods, commas and question marks belong from pauses and the way sentences are spoken. Many systems also reformat what they hear, so “thirty thousand dollars” becomes “$30,000”. Both are guesses, so double-check amounts, dates and phone numbers. Whether filler words like “um” and “uh” are kept or removed depends on the service and its settings.

What should you watch for when handling recordings?

Recordings and transcripts often contain information about people: names, contact details, what someone asked for help with. Japan’s privacy guidelines, for example, list a voice recording that can identify a specific person (because their name is spoken, say) as personal information. Rules differ from country to country, so check the ones that apply to you.

• Tell everyone in the conversation before you start recording
• Find out where the audio is processed: on the device or by an online service
• Check the terms of service and data policy to see whether recordings or transcripts are used to train models, and how long they’re kept
• Limit who can open recordings and transcripts, and delete them when you no longer need them
• Don’t say passwords or card numbers out loud while recording

Captions need extra care. The W3C, which develops web standards, says in its guidance on captions and subtitles that automatically generated captions don’t meet user needs or accessibility requirements unless they’re confirmed to be fully accurate. If you caption the videos in your online course, treat automatic captions as a first draft and have a person correct them before you publish.

Summary: tips for getting good results

Speech recognition turns spoken words into text, and how well it does depends a lot on how you record and speak.

1. Speak somewhere quiet, with the microphone close to your mouth.
2. In meetings, take turns, and tell everyone up front that you’re recording.
3. Add names and specialist terms in advance if the service allows it.
4. Check amounts, dates and names once the text is ready.
5. Decide where recordings and transcripts are kept, for how long, and who can see them.