notebooks/211-speech-to-text/211-speech-to-text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf874fc4",
   "metadata": {},
   "source": [
    "# Speech to Text with OpenVINO™\n",
    "\n",
    "This tutorial demonstrates speech-to-text recognition with OpenVINO.\n",
    "\n",
    "This tutorial uses the [quartznet 15x5](https://docs.openvino.ai/2021.4/omz_models_model_quartznet_15x5_en.html) model. QuartzNet performs automatic speech recognition. Its design is based on the Jasper architecture, which is a convolutional model trained with Connectionist Temporal Classification (CTC) loss. The model is available from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import IPython.display as ipd\n",
    "import librosa\n",
    "import librosa.display\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy\n",
    "from openvino.runtime import Core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings\n",
    "\n",
    "In this part, all variables used in the notebook are set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_folder = \"model\"\n",
    "download_folder = \"output\"\n",
    "data_folder = \"data\"\n",
    "\n",
    "precision = \"FP16\"\n",
    "model_name = \"quartznet-15x5-en\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and Convert Public Model\n",
    "If it is your first run, models will be downloaded and converted here. It my take a few minutes. Use `omz_downloader` and `omz_converter`, which are command-line tools from the `openvino-dev` package. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d946d37",
   "metadata": {},
   "source": [
    "### Download Model\n",
    "\n",
    "The `omz_downloader` tool automatically creates a directory structure and downloads the selected model. This step is skipped if the model is already downloaded. The selected model comes from the public directory, which means it must be converted into OpenVINO Intermediate Representation (OpenVINO IR)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if a model is already downloaded (to the download directory).\n",
    "path_to_model_weights = Path(f'{download_folder}/public/{model_name}/models')\n",
    "downloaded_model_file = list(path_to_model_weights.glob('*.pth'))\n",
    "\n",
    "if not path_to_model_weights.is_dir() or len(downloaded_model_file) == 0:\n",
    "    download_command = f\"omz_downloader --name {model_name} --output_dir {download_folder} --precision {precision}\"\n",
    "    ! $download_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c0a5cdb",
   "metadata": {},
   "source": [
    "### Convert Model\n",
    "\n",
    "The `omz_converter` tool is needed to convert pre-trained `PyTorch` model to ONNX model format, which is further converted to OpenVINO IR format. Both stages of conversion are handled by using `omz_converter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cfaf7ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if a model is already converted (in the model directory).\n",
    "path_to_converted_weights = Path(f'{model_folder}/public/{model_name}/{precision}/{model_name}.bin')\n",
    "\n",
    "if not path_to_converted_weights.is_file():\n",
    "    convert_command = f\"omz_converter --name {model_name} --precisions {precision} --download_dir {download_folder} --output_dir {model_folder}\"\n",
    "    ! $convert_command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Audio Processing\n",
    "\n",
    "Now that the model is converted, load an audio file. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c4ad393",
   "metadata": {},
   "source": [
    "### Define constants\n",
    "\n",
    "First, locate an audio file and define the alphabet used by the model. This tutorial uses the Latin alphabet beginning with a space symbol and ending with a blank symbol. In this case it will be `~`, but that could be any other character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio_file_name = \"edge_to_cloud.ogg\"\n",
    "alphabet = \" abcdefghijklmnopqrstuvwxyz'~\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47592c21",
   "metadata": {},
   "source": [
    "### Available Audio Formats\n",
    "\n",
    "There are multiple supported audio formats that can be used with the model: \n",
    "\n",
    "AIFF, AU, AVR, CAF, FLAC, HTK, SVX, MAT4, MAT5, MPC2K, OGG, PAF, PVF, RAW, RF64, SD2, SDS, IRCAM, VOC, W64, WAV, NIST, WAVEX, WVE, XI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aa408c2",
   "metadata": {},
   "source": [
    "### Load Audio File\n",
    "\n",
    "Load the file after checking a file extension. Pass `sr` (stands for a `sampling rate`) as an additional parameter. The model supports files with a `sampling rate` of 16 kHz."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audio, sampling_rate = librosa.load(path=f'{data_folder}/{audio_file_name}', sr=16000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5271feee",
   "metadata": {},
   "source": [
    "Now, you can play your audio file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a8db1c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "ipd.Audio(audio, rate=sampling_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4fd06e0",
   "metadata": {},
   "source": [
    "### Visualise Audio File\n",
    "\n",
    "You can visualize how your audio file presents on a wave plot and spectrogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecbd9d4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "librosa.display.waveshow(y=audio, sr=sampling_rate, max_points=50000, x_axis='time', offset=0.0);\n",
    "plt.show()\n",
    "specto_audio = librosa.stft(audio)\n",
    "specto_audio = librosa.amplitude_to_db(np.abs(specto_audio), ref=np.max)\n",
    "print(specto_audio.shape)\n",
    "librosa.display.specshow(specto_audio, sr=sampling_rate, x_axis='time', y_axis='hz');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0564a2d8",
   "metadata": {},
   "source": [
    "### Change Type of Data\n",
    "\n",
    "The file loaded in the previous step may contain data in `float` type with a range of values between -1 and 1. To generate a viable input, multiply each value by the max value of `int16` and convert it to `int16` type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eb5cbe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "if max(np.abs(audio)) <= 1:\n",
    "    audio = (audio * (2**15 - 1))\n",
    "audio = audio.astype(np.int16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "666a47cd",
   "metadata": {},
   "source": [
    "### Convert Audio to Mel Spectrum\n",
    "\n",
    "Next, convert the pre-pre-processed audio to [Mel Spectrum](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53). For more information on why it needs to be done, refer to [this article](https://towardsdatascience.com/audio-deep-learning-made-simple-part-2-why-mel-spectrograms-perform-better-aad889a93505)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def audio_to_mel(audio, sampling_rate):\n",
    "    assert sampling_rate == 16000, \"Only 16 KHz audio supported\"\n",
    "    preemph = 0.97\n",
    "    preemphased = np.concatenate([audio[:1], audio[1:] - preemph * audio[:-1].astype(np.float32)])\n",
    "\n",
    "    # Calculate the window length.\n",
    "    win_length = round(sampling_rate * 0.02)\n",
    "\n",
    "    # Based on the previously calculated window length, run short-time Fourier transform.\n",
    "    spec = np.abs(librosa.core.spectrum.stft(preemphased, n_fft=512, hop_length=round(sampling_rate * 0.01),\n",
    "                  win_length=win_length, center=True, window=scipy.signal.windows.hann(win_length), pad_mode='reflect'))\n",
    "\n",
    "    # Create mel filter-bank, produce transformation matrix to project current values onto Mel-frequency bins.\n",
    "    mel_basis = librosa.filters.mel(sr=sampling_rate, n_fft=512, n_mels=64, fmin=0.0, fmax=8000.0, htk=False)\n",
    "    return mel_basis, spec\n",
    "\n",
    "\n",
    "def mel_to_input(mel_basis, spec, padding=16):\n",
    "    # Convert to a logarithmic scale.\n",
    "    log_melspectrum = np.log(np.dot(mel_basis, np.power(spec, 2)) + 2 ** -24)\n",
    "\n",
    "    # Normalize the output.\n",
    "    normalized = (log_melspectrum - log_melspectrum.mean(1)[:, None]) / (log_melspectrum.std(1)[:, None] + 1e-5)\n",
    "\n",
    "    # Calculate padding.\n",
    "    remainder = normalized.shape[1] % padding\n",
    "    if remainder != 0:\n",
    "        return np.pad(normalized, ((0, 0), (0, padding - remainder)))[None]\n",
    "    return normalized[None]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d86046e",
   "metadata": {},
   "source": [
    "### Run Conversion from Audio to Mel Format\n",
    "\n",
    "In this step, convert a current audio file into [Mel scale](https://en.wikipedia.org/wiki/Mel_scale)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mel_basis, spec = audio_to_mel(audio=audio.flatten(), sampling_rate=sampling_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e3276fa",
   "metadata": {},
   "source": [
    "### Visualise Mel Spectogram\n",
    "\n",
    "For more information about Mel spectrogram, refer to this [article](https://towardsdatascience.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0). The first image visualizes Mel frequency spectrogram, the second one presents filter bank for converting Hz to Mels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef57088",
   "metadata": {},
   "outputs": [],
   "source": [
    "librosa.display.specshow(data=spec, sr=sampling_rate, x_axis='time', y_axis='log');\n",
    "plt.show();\n",
    "librosa.display.specshow(data=mel_basis, sr=sampling_rate, x_axis='linear');\n",
    "plt.ylabel('Mel filter');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "503f1778",
   "metadata": {},
   "source": [
    "### Adjust Mel scale to Input\n",
    "\n",
    "Before reading the network, make sure that the input is ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a63159b",
   "metadata": {},
   "outputs": [],
   "source": [
    "audio = mel_to_input(mel_basis=mel_basis, spec=spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4aec612",
   "metadata": {},
   "source": [
    "## Load the Model\n",
    "\n",
    "Now, you can read and load the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d5dc5c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = Core()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3fc33e",
   "metadata": {},
   "source": [
    "You may run the network on multiple devices. By default, it will load the model on CPU (you can choose manually CPU, GPU, MYRIAD, etc.) or let the engine choose the best available device (AUTO).\n",
    "\n",
    "To list all available devices that can be used, run `print(ie.available_devices)` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a69236c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ie.available_devices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57d2f27",
   "metadata": {},
   "source": [
    "To change the device used for your network, change value of `device_name` variable to one of the values listed by `print()` in the cell above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ie.read_model(\n",
    "    model=f\"{model_folder}/public/{model_name}/{precision}/{model_name}.xml\"\n",
    ")\n",
    "model_input_layer = model.input(0)\n",
    "shape = model_input_layer.partial_shape\n",
    "shape[2] = -1\n",
    "model.reshape({model_input_layer: shape})\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"CPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a566de49",
   "metadata": {},
   "source": [
    "### Do Inference\n",
    "\n",
    "Everything is set up. Now, the only thing that remains is passing input to the previously loaded network and running inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_layer_ir = compiled_model.output(0)\n",
    "\n",
    "character_probabilities = compiled_model([audio])[output_layer_ir]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc761843",
   "metadata": {},
   "source": [
    "### Read Output\n",
    "\n",
    "After inference, you need to reach out the output. The default output format for `quartznet 15x5` are per-frame probabilities (after LogSoftmax) for every symbol in the alphabet, name - output, shape - 1x64x29, output data format is BxNxC, where:\n",
    "\n",
    "* B - batch size\n",
    "* N - number of audio frames\n",
    "* C - alphabet size, including the Connectionist Temporal Classification (CTC) blank symbol\n",
    "\n",
    "You need to make it in a more human-readable format. To do this you, use a symbol with the highest probability. When you hold a list of indexes that are predicted to have the highest probability, due to limitations given by [Connectionist Temporal Classification Decoding](https://towardsdatascience.com/beam-search-decoding-in-ctc-trained-neural-networks-5a889a3d85a7) you will remove concurrent symbols and then remove all the blanks.\n",
    "\n",
    "The last step is getting symbols from corresponding indexes in charlist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce9e3604",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove unnececery dimension\n",
    "character_probabilities = np.squeeze(character_probabilities)\n",
    "\n",
    "# Run argmax to pick most possible symbols\n",
    "character_probabilities = np.argmax(character_probabilities, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "477d8b66",
   "metadata": {},
   "source": [
    "### Implementation of Decoding\n",
    "\n",
    "To decode previously explained output, you need the [Connectionist Temporal Classification (CTC) decode](https://towardsdatascience.com/beam-search-decoding-in-ctc-trained-neural-networks-5a889a3d85a7) function. This solution will remove consecutive letters from the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cfd8a5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ctc_greedy_decode(predictions):\n",
    "    previous_letter_id = blank_id = len(alphabet) - 1\n",
    "    transcription = list()\n",
    "    for letter_index in predictions:\n",
    "        if previous_letter_id != letter_index != blank_id:\n",
    "            transcription.append(alphabet[letter_index])\n",
    "        previous_letter_id = letter_index\n",
    "    return ''.join(transcription)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e9d9b9",
   "metadata": {},
   "source": [
    "### Run Decoding and Print Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcription = ctc_greedy_decode(character_probabilities)\n",
    "print(transcription)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
  },
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}