multilingual_speech_recognition.en.md 23 KB
comments: true
Multilingual Speech Recognition pipeline Tutorial
1. Introduction to Multilingual Speech Recognition pipeline
Speech recognition is an advanced tool that can automatically convert spoken languages into corresponding text or commands. This technology plays an important role in various fields such as intelligent customer service, voice assistants, and meeting records. Multilingual speech recognition supports automatic language detection and recognition of multiple languages.
The inference time only includes the model inference time and does not include the time for pre- or post-processing.
Multilingual Speech Recognition Models (Optional):
| Model | Model Download Link | Training Data | Model Storage Size (MB) | Word Error Rate | Introduction |
|---|---|---|---|---|---|
| whisper_large | whisper_large | 680kh | 5800 | 2.7 (Librispeech) | Whisper is a multilingual automatic speech recognition model developed by OpenAI, known for its high precision and robustness. It features an end-to-end architecture and can handle noisy audio environments, making it suitable for applications such as voice assistants and real-time subtitles. |
| whisper_medium | whisper_medium | 680kh | 2900 | - | |
| whisper_small | whisper_small | 680kh | 923 | - | |
| whisper_base | whisper_base | 680kh | 277 | - | |
| whisper_tiny | whisper_tiny | 680kh | 145 | - |
multilingual_speech_recognition pipeline object is instantiated through create_pipeline(). The specific parameter descriptions are as follows:
| Parameter | Parameter Description | Parameter Type | Default Value | |
|---|---|---|---|---|
pipeline |
The name of the pipeline or the path to the pipeline configuration file. If it is the pipeline name, it must be a pipeline supported by PaddleX. | str |
None |
|
device |
The inference device for the pipeline. It supports specifying the specific card number of the GPU, such as "gpu:0", the specific card number of other hardware, such as "npu:0", and the CPU, such as "cpu". | str |
gpu:0 |
|
use_hpip |
Whether to enable the high-performance inference plugin. If set to None, the setting from the configuration file or config will be used. Not supported for now. |
bool |
None | None |
hpi_config |
High-performance inference configuration. Not supported for now. | dict | None |
None | None |
predict() method of the multilingual_speech_recognition pipeline object is called to perform inference and prediction. This method will return a generator. Below are the parameters and their descriptions for the predict() method:
| Parameter | Parameter Description | Parameter Type | Options | Default Value |
|---|---|---|---|---|
input |
Data to be predicted | str |
|
None |
| Method | Description | Parameter | Parameter Type | Parameter Description | Default Value |
|---|---|---|---|---|---|
print() |
Print the result to the terminal | format_json |
bool |
Whether to format the output content using JSON indentation |
True |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True |
False |
||
save_to_json() |
Save the result as a JSON file | save_path |
str |
Path to save the file. When it is a directory, the saved file name is consistent with the input file type naming | None |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False will retain the original characters. Effective only when format_json is True |
False |
| Attribute | Attribute Description |
|---|---|
json |
Get the predicted json format result |
multilingual_speech_recognition pipeline, you can directly modify the parameters in the configuration file and load the configuration file for prediction. Additionally, CLI prediction also supports passing in a configuration file, simply specify the path of the configuration file with --pipeline.
Multilingual Service Call Examples
Python
import base64
import requests
API_URL = "http://localhost:8080/multilingual-speech-recognition"
audio_path = "./zh.wav"
with open(audio_path, "rb") as file:
audio_bytes = file.read()
audio_data = base64.b64encode(audio_bytes).decode("ascii")
payload = {"audio": audio_data}
response = requests.post(API_URL, json=payload)
assert response.status_code == 200
result = response.json()["result"]
print(result)
📱 On-Device Deployment: Edge deployment is a method of placing computing and data processing capabilities directly on the user's device, allowing it to process data locally without relying on remote servers. PaddleX supports deploying models on edge devices such as Android. For detailed procedures on edge deployment, please refer to the [PaddleX On-Device Deployment Guide](../../../pipeline_deploy/on_device_deployment.en.md). You can choose the appropriate method to deploy the model pipeline according to your needs and proceed with subsequent AI application integration. ## 3. Development Integration/Deployment If the pipeline meets your requirements for inference speed and accuracy, you can directly proceed with development integration/deployment. If you need to apply the pipeline directly in your Python project, you can refer to the example code in [2.2.2 Python Script Method](#222-python脚本方式集成). In addition, PaddleX also provides three other deployment methods, which are detailed as follows: 🚀 High-Performance Inference: In actual production environments, many applications have strict performance requirements for deployment strategies, especially in terms of response speed, to ensure the efficient operation of the system and the smoothness of the user experience. To this end, PaddleX provides a high-performance inference plugin, which aims to deeply optimize the performance of model inference and pre/post-processing to achieve significant acceleration of the end-to-end process. For detailed high-performance inference procedures, please refer to the [PaddleX High-Performance Inference Guide](../../../pipeline_deploy/high_performance_inference.en.md). ☁️ Serving Deployment: Serving Deployment is a common deployment form in actual production environments. By encapsulating inference functions as services, clients can access these services through network requests to obtain inference results. PaddleX supports multiple pipeline serving deployment solutions. For detailed pipeline serving deployment procedures, please refer to the [PaddleX Serving Deployment Guide](../../../pipeline_deploy/serving.en.md). Below are the API references for basic serving deployment and examples of multi-language service calls:
API Reference
For the main operations provided by the service:
- The HTTP request method is POST.
- Both the request body and response body are JSON data (JSON objects).
- When the request is processed successfully, the response status code is
200, and the properties of the response body are as follows:
| Name | Type | Meaning |
|---|---|---|
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Fixed as 0. |
errorMsg |
string |
Error message. Fixed as "Success". |
result |
object |
The result of the operation. |
- When the request is not processed successfully, the properties of the response body are as follows:
| Name | Type | Meaning |
|---|---|---|
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Same as the response status code. |
errorMsg |
string |
Error message. |
The main operations provided by the service are as follows:
infer
Perform multilingual speech recognition on audio.
POST /multilingual-speech-recognition
- The properties of the request body are as follows:
| Name | Type | Meaning | Required |
|---|---|---|---|
audio |
string |
The URL or path of the audio file accessible by the server. | Yes |
- When the request is processed successfully, the
resultof the response body has the following properties:
| Name | Type | Meaning |
|---|---|---|
text |
string |
The text result of speech recognition. |
segments |
array |
The result text with timestamps. |
language |
string |
The recognized language. |
Each element in segments is an object with the following properties:
| Name | Type | Meaning |
|---|---|---|
id |
integer |
The ID of the audio segment. |
seek |
integer |
The pointer of the audio segment. |
start |
number |
The start time of the audio segment. |
end |
number |
The end time of the audio segment. |
text |
string |
The recognized text of the audio segment. |
tokens |
array |
The token IDs of the audio segment. |
temperature |
number |
The speed change ratio. |
avgLogProb |
number |
The average log probability. |
compressionRatio |
number |
The compression ratio. |
noSpeechProb |
number |
The probability of no speech. |
Example of Multilingual Service Invocation
Python
import requests
API_URL = "http://localhost:8080/multilingual-speech-recognition" # Service URL
audio_path = "./zh.wav"
payload = {"audio": audio_path}
Invoke API
response = requests.post(API_URL, json=payload)
Process API response
assert response.status_code == 200
result = response.json()["result"]
print(result)
📱 On-Device Deployment: Edge deployment is a method that places computational and data processing capabilities directly on user devices, allowing them to process data without relying on remote servers. PaddleX supports deploying models on edge devices such as Android. For detailed procedures, please refer to the PaddleX On-Device Deployment Guide. You can choose the appropriate deployment method based on your needs to integrate the model into your pipeline and proceed with subsequent AI application integration.
4. Custom Development
If the default model weights provided by the general video classification pipeline are not satisfactory in terms of accuracy or speed for your specific scenario, you can attempt to fine-tune the existing model using your own domain-specific or application-specific data to improve the recognition performance of the general video classification pipeline in your scenario.
4.1 Model Fine-Tuning
Since the general video classification pipeline only includes a video classification module, if the performance of the pipeline is not up to expectations, you can analyze the videos with poor recognition and refer to the corresponding fine-tuning tutorial links in the table below for model fine-tuning.
| Scenario | Fine-Tuning Module | Fine-Tuning Reference Link |
|---|---|---|
| Inaccurate video classification | Video Classification Module | Link |
4.2 Model Application
After completing the fine-tuning with your private dataset, you will obtain the local model weight file.
If you need to use the fine-tuned model weights, simply modify the pipeline configuration file by replacing the path to the fine-tuned model weights with the corresponding location in the pipeline configuration file:
from paddlex import create_pipeline
pipeline = create_pipeline(pipeline="./my_path/multilingual_speech_recognition.yaml")
output = pipeline.predict(input="zh.wav")
for res in output:
res.print()
res.save_to_json("./output/")
Subsequently, refer to the command-line method or Python script method in the local experience to load the modified pipeline configuration file.
5. Multi-Hardware Support
PaddleX supports a variety of mainstream hardware devices, including NVIDIA GPU, Kunlunxin XPU, Ascend NPU, and Cambricon MLU. Simply modify the --device parameter to seamlessly switch between different hardware devices.
For example, if you use Ascend NPU for video classification in the pipeline, the Python command used is: