PaddleX/docs/pipeline_usage/tutorials/speech_pipelines/multilingual_speech_recognition.en.md

---

comments: true

Multilingual Speech Recognition pipeline Tutorial

1. Introduction to Multilingual Speech Recognition pipeline

Speech recognition is an advanced tool that can automatically convert spoken languages into corresponding text or commands. This technology plays an important role in various fields such as intelligent customer service, voice assistants, and meeting records. Multilingual speech recognition supports automatic language detection and recognition of multiple languages.

Multilingual Speech Recognition Models (Optional):

Model	Model Download Link	Training Data	Model Size	Word Error Rate	Introduction
whisper_large	whisper_large	680kh	5.8G	2.7 (Librispeech)	Whisper is a multilingual automatic speech recognition model developed by OpenAI, known for its high precision and robustness. It features an end-to-end architecture and can handle noisy audio environments, making it suitable for applications such as voice assistants and real-time subtitles.
whisper_medium	whisper_medium	680kh	2.9G	-
whisper_small	whisper_small	680kh	923M	-
whisper_base	whisper_base	680kh	277M	-
whisper_tiny	whisper_tiny	680kh	145M	-

## 2. Quick Start PaddleX supports experiencing the multilingual speech recognition pipeline locally using the command line or Python. Before using the multilingual speech recognition pipeline locally, please ensure that you have completed the installation of the PaddleX wheel package according to the [PaddleX Local Installation Guide](../../../installation/installation.en.md). If you wish to selectively install dependencies, please refer to the relevant instructions in the installation guide. The dependency group corresponding to this pipeline is `speech`. ### 2.1 Local Experience #### 2.1.1 Command Line Experience You can quickly experience the effect of the document image preprocessing pipeline with a single command. Use the [example audio](https://paddlespeech.bj.bcebos.com/PaddleAudio/zh.wav) and replace `--input` with the local path for prediction. ```bash paddlex --pipeline multilingual_speech_recognition \ --input zh.wav \ --save_path ./output \ --device gpu:0 ``` The relevant parameter descriptions can be found in the parameter descriptions in [2.1.2 Integration via Python Script](). After running, the result will be printed to the terminal, as follows: ```bash {'input_path': 'zh.wav', 'result': {'text': '我认为跑步最重要的就是给我带来了身体健康', 'segments': [{'id': 0, 'seek': 0, 'start': 0.0, 'end': 2.0, 'text': '我认为跑步最重要的就是', 'tokens': [50364, 1654, 7422, 97, 13992, 32585, 31429, 8661, 24928, 1546, 5620, 50464, 50464, 49076, 4845, 99, 34912, 19847, 29485, 44201, 6346, 115, 50564], 'temperature': 0, 'avg_logprob': -0.22779104113578796, 'compression_ratio': 0.28169014084507044, 'no_speech_prob': 0.026114309206604958}, {'id': 1, 'seek': 200, 'start': 2.0, 'end': 31.0, 'text': '给我带来了身体健康', 'tokens': [50364, 49076, 4845, 99, 34912, 19847, 29485, 44201, 6346, 115, 51814], 'temperature': 0, 'avg_logprob': -0.21976988017559052, 'compression_ratio': 0.23684210526315788, 'no_speech_prob': 0.009023111313581467}], 'language': 'zh'}} ``` The explanation of the result parameters can refer to the result explanation in [2.1.2 Integration with Python Script](#212-integration-with-python-script). #### 2.1.2 Integration with Python Script The above command line is for quickly experiencing and viewing the effect. Generally speaking, in a project, it is often necessary to integrate through code. You can complete the rapid inference of the pipeline with just a few lines of code. The inference code is as follows: ```python from paddlex import create_pipeline pipeline = create_pipeline(pipeline="multilingual_speech_recognition") output = pipeline.predict(input="zh.wav") for res in output: res.print() res.save_to_json(save_path="./output/") ``` In the above Python script, the following steps are executed: (1) The multilingual_speech_recognition pipeline object is instantiated through create_pipeline(). The specific parameter descriptions are as follows:

Parameter	Parameter Description	Parameter Type	Default Value
pipeline	The name of the pipeline or the path to the pipeline configuration file. If it is the pipeline name, it must be a pipeline supported by PaddleX.	str	None
device	The inference device for the pipeline. It supports specifying the specific card number of the GPU, such as "gpu:0", the specific card number of other hardware, such as "npu:0", and the CPU, such as "cpu".	str	gpu:0

(2) The predict() method of the multilingual_speech_recognition pipeline object is called to perform inference and prediction. This method will return a generator. Below are the parameters and their descriptions for the predict() method:

Parameter	Parameter Description	Parameter Type	Options	Default Value
input	Data to be predicted	str	File path, such as the local path of an audio file: /root/data/audio.wav URL link, such as the network URL of an audio file: Example	None
device	The inference device for the pipeline	str|None	CPU: such as cpu indicates using the CPU for inference; GPU: such as gpu:0 indicates using the first GPU for inference; NPU: such as npu:0 indicates using the first NPU for inference; XPU: such as xpu:0 indicates using the first XPU for inference; MLU: such as mlu:0 indicates using the first MLU for inference; DCU: such as dcu:0 indicates using the first DCU for inference; None: If set to None, the default value initialized for the pipeline will be used. During initialization, the local GPU device 0 will be prioritized. If it is not available, the CPU device will be used.	None

(3) Process the prediction results. The prediction result for each sample is of the `dict` type and supports operations such as printing, saving as an image, and saving as a `json` file:

Method	Description	Parameter	Parameter Type	Parameter Description	Default Value
print()	Print the result to the terminal	format_json	bool	Whether to format the output content using JSON indentation	True
		indent	int	Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True	4
		ensure_ascii	bool	Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True	False
save_to_json()	Save the result as a JSON file	save_path	str	Path to save the file. When it is a directory, the saved file name is consistent with the input file type naming	None
		indent	int	Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True	4
		ensure_ascii	bool	Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True	False

- Calling the `print()` method will print the result to the terminal, with the printed content explained as follows: - `input_path`: The path where the input audio is stored - `result`: Recognition result - `text`: The text result of speech recognition - `segments`: The result text with timestamps * `id`: ID * `seek`: Audio segment pointer * `start`: Segment start time * `end`: Segment end time * `text`: Text recognized in the segment * `tokens`: Token IDs of the segment text * `temperature`: Speed variation ratio * `avg_logprob`: Average log probability * `compression_ratio`: Compression ratio * `no_speech_prob`: Non-speech probability - `language`: Recognized language - Calling the `save_to_json()` method will save the above content to the specified `save_path`. If specified as a directory, the saved path will be `save_path/{your_audio_basename}.json`; if specified as a file, it will be saved directly to that file. Since JSON files do not support saving numpy arrays, the `numpy.array` types will be converted to lists. * Additionally, it also supports obtaining visualized images and prediction results through attributes, as follows:

Attribute	Attribute Description
json	Get the predicted json format result

- The prediction result obtained by the `json` attribute is a dict type of data, with content consistent with the content saved by calling the `save_to_json()` method. In addition, you can obtain the multilingual_speech_recognition pipeline configuration file and load the configuration file for prediction. You can execute the following command to save the result in `my_path`: ``` paddlex --get_pipeline_config multilingual_speech_recognition --save_path ./my_path ``` If you have obtained the configuration file, you can customize the settings for the `multilingual_speech_recognition` pipeline. Simply modify the value of the `pipeline` parameter in the `create_pipeline` method to the path of the pipeline configuration file. An example is as follows: For example, if your configuration file is saved at `./my_path/multilingual_speech_recognition.yaml`, you just need to execute: ```python from paddlex import create_pipeline pipeline = create_pipeline(pipeline="multilingual_speech_recognition") output = pipeline.predict(input="zh.wav") for res in output: res.print() res.save_to_json(save_path="./output/") ``` Note: The parameters in the configuration file are the initialization parameters for the pipeline. If you want to change the initialization parameters of the multilingual_speech_recognition pipeline, you can directly modify the parameters in the configuration file and load the configuration file for prediction. Additionally, CLI prediction also supports passing in a configuration file, simply specify the path of the configuration file with --pipeline.

Multilingual Service Call Examples

Python

import base64
import requests

API_URL = "http://localhost:8080/multilingual-speech-recognition"
audio_path = "./zh.wav"

with open(audio_path, "rb") as file:
    audio_bytes = file.read()
    audio_data = base64.b64encode(audio_bytes).decode("ascii")

payload = {"audio": audio_data}

response = requests.post(API_URL, json=payload)

assert response.status_code == 200

result = response.json()["result"]
print(result)

📱 Edge Deployment: Edge deployment is a method of placing computing and data processing capabilities directly on the user's device, allowing it to process data locally without relying on remote servers. PaddleX supports deploying models on edge devices such as Android. For detailed procedures on edge deployment, please refer to the [PaddleX Edge Deployment Guide](../../../pipeline_deploy/edge_deploy.en.md). You can choose the appropriate method to deploy the model pipeline according to your needs and proceed with subsequent AI application integration. ## 3. Development Integration/Deployment If the pipeline meets your requirements for inference speed and accuracy, you can directly proceed with development integration/deployment. If you need to apply the pipeline directly in your Python project, you can refer to the example code in [2.2.2 Python Script Method](#222-python脚本方式集成). In addition, PaddleX also provides three other deployment methods, which are detailed as follows: 🚀 High-Performance Inference: In actual production environments, many applications have strict performance requirements for deployment strategies, especially in terms of response speed, to ensure the efficient operation of the system and the smoothness of the user experience. To this end, PaddleX provides a high-performance inference plugin, which aims to deeply optimize the performance of model inference and pre/post-processing to achieve significant acceleration of the end-to-end process. For detailed high-performance inference procedures, please refer to the [PaddleX High-Performance Inference Guide](../../../pipeline_deploy/high_performance_inference.en.md). ☁️ Serving Deployment: Serving Deployment is a common deployment form in actual production environments. By encapsulating inference functions as services, clients can access these services through network requests to obtain inference results. PaddleX supports multiple pipeline serving deployment solutions. For detailed pipeline serving deployment procedures, please refer to the [PaddleX Serving Deployment Guide](../../../pipeline_deploy/serving.en.md). Below are the API references for basic serving deployment and examples of multi-language service calls:

API Reference

For the main operations provided by the service:

• The HTTP request method is POST.
• Both the request body and response body are JSON data (JSON objects).
• When the request is processed successfully, the response status code is 200, and the properties of the response body are as follows:

Name	Type	Meaning
logId	string	The UUID of the request.
errorCode	integer	Error code. Fixed as 0.
errorMsg	string	Error message. Fixed as "Success".
result	object	The result of the operation.

• When the request is not processed successfully, the properties of the response body are as follows:

Name	Type	Meaning
logId	string	The UUID of the request.
errorCode	integer	Error code. Same as the response status code.
errorMsg	string	Error message.

The main operations provided by the service are as follows:

• infer

Perform multilingual speech recognition on audio.

POST /multilingual-speech-recognition

• The properties of the request body are as follows:

Name	Type	Meaning	Required
audio	string	The URL or path of the audio file accessible by the server.	Yes

• When the request is processed successfully, the result of the response body has the following properties:

Name	Type	Meaning
text	string	The text result of speech recognition.
segments	array	The result text with timestamps.
language	string	The recognized language.

Each element in segments is an object with the following properties:

Name	Type	Meaning
id	integer	The ID of the audio segment.
seek	integer	The pointer of the audio segment.
start	number	The start time of the audio segment.
end	number	The end time of the audio segment.
text	string	The recognized text of the audio segment.
tokens	array	The token IDs of the audio segment.
temperature	number	The speed change ratio.
avgLogProb	number	The average log probability.
compressionRatio	number	The compression ratio.
noSpeechProb	number	The probability of no speech.

Example of Multilingual Service Invocation

Python

import requests


API_URL = "http://localhost:8080/multilingual-speech-recognition" # Service URL
audio_path = "./zh.wav"

payload = {"audio": audio_path}

Invoke API

response = requests.post(API_URL, json=payload)

Process API response

assert response.status_code == 200
result = response.json()["result"]
print(result)

📱 Edge Deployment: Edge deployment is a method that places computational and data processing capabilities directly on user devices, allowing them to process data without relying on remote servers. PaddleX supports deploying models on edge devices such as Android. For detailed procedures, please refer to the PaddleX Edge Deployment Guide. You can choose the appropriate deployment method based on your needs to integrate the model into your pipeline and proceed with subsequent AI application integration.

4. Custom Development

If the default model weights provided by the general video classification pipeline are not satisfactory in terms of accuracy or speed for your specific scenario, you can attempt to fine-tune the existing model using your own domain-specific or application-specific data to improve the recognition performance of the general video classification pipeline in your scenario.

4.1 Model Fine-Tuning

Since the general video classification pipeline only includes a video classification module, if the performance of the pipeline is not up to expectations, you can analyze the videos with poor recognition and refer to the corresponding fine-tuning tutorial links in the table below for model fine-tuning.

Scenario	Fine-Tuning Module	Fine-Tuning Reference Link
Inaccurate video classification	Video Classification Module	Link

4.2 Model Application

After completing the fine-tuning with your private dataset, you will obtain the local model weight file.

If you need to use the fine-tuned model weights, simply modify the pipeline configuration file by replacing the path to the fine-tuned model weights with the corresponding location in the pipeline configuration file:

from paddlex import create_pipeline
pipeline = create_pipeline(pipeline="./my_path/multilingual_speech_recognition.yaml")
output = pipeline.predict(input="zh.wav")
for res in output:
    res.print()
    res.save_to_json("./output/")

Subsequently, refer to the command-line method or Python script method in the local experience to load the modified pipeline configuration file.

5. Multi-Hardware Support

PaddleX supports a variety of mainstream hardware devices, including NVIDIA GPU, Kunlunxin XPU, Ascend NPU, and Cambricon MLU. Simply modify the --device parameter to seamlessly switch between different hardware devices.

For example, if you use Ascend NPU for video classification in the pipeline, the Python command used is: