#292: Extract Text From Audio Files With Whisper

With Vosk we got last week some good results with the transcription of our audio file. It missed the capitalisation and punctuation, but it got the text itself right. In this post we try the same, but this time with the Whisper library from OpenAI.

What is Whisper?

Whisper is an open-source neural network developed by OpenAI that can transcribe spoken language into written text. It supports multiple languages and is robust against different accents, background noise, and recording conditions. Whisper was trained on a large dataset of multilingual and multitask supervised data collected from the web, making it capable of handling real-world, noisy audio with impressive accuracy. Whisper's code and model weights are released under the MIT License.

Installation

We can install the Python package for Whisper with this command:

uv pip install -U openai-whisper

This is only the first part. To be able to access the different audio formats, we also need the FFmpeg library. We can download the pre-build binaries for Windows or run this command on Linux to install FFmpeg:

sudo apt update && sudo apt install ffmpeg

Select a model

We now need to decide on what model we want to use. We can select any of the model in this table:

Size	Parameters	English-only model	Multilingual model	Required VRAM	Relative speed
tiny	39 M	tiny.en	tiny	~1 GB	~10x
base	74 M	base.en	base	~1 GB	~7x
small	244 M	small.en	small	~2 GB	~4x
medium	769 M	medium.en	medium	~5 GB	~2x
large	1550 M	N/A	large	~10 GB	1x
turbo	809 M	N/A	turbo	~6 GB	~8x

We only need to know the name of the model to use it in our script. The Whisper package will download it at the first run and sets up everything for us.

Transcribe the audio file

To transcribe the audio file we generated in the last post, we can use these few lines of code:

import whisper

model = whisper.load_model("tiny", device="cpu")
result = model.transcribe("audio_demo.wav")
print(result["text"])

This uses the tiny model and produces this output:

Python Friday is a weekly blog about the Python programming language. It covers a wide range of topics from web development to data visualization and artificial intelligence.

As a reminder, the text to generate the audio file was this one: