Video detection tasks are a critical component of computer vision systems, focusing on identifying and locating objects or events within video sequences. Video detection involves decomposing the video into individual frame sequences and then analyzing these frames to recognize detected objects or actions, such as detecting pedestrians in surveillance videos or identifying specific activities like "running," "jumping," or "playing guitar" in sports or entertainment videos.
The output of the video detection module includes bounding boxes and class labels for each detected object or event. This information can be used by other modules or systems for further analysis, such as tracking the movement of detected objects, generating alerts, or compiling statistics for decision-making processes. Therefore, video detection plays an important role in various applications ranging from security surveillance and autonomous driving to sports analytics and content moderation.
| Model | Model Download Link | Frame-mAP (@ IoU 0.5) | Model Storage Size (M) | Description |
|---|---|---|---|---|
| YOWO | Inference Model/训练模型 | 80.94 | 462.891M | YOWO is a single-stage network with two branches. One branch extracts spatial features of key frames (i.e., the current frame) through a 2D-CNN, while the other branch captures spatiotemporal features of a clip composed of previous frames using a 3D-CNN. To accurately aggregate these features, YOWO employs a channel fusion and attention mechanism to maximize the utilization of inter-channel dependencies. Finally, the fused features are used for frame-level detection. |
Note: The above accuracy metrics refer to Frame-mAP (@ IoU 0.5) Accuracy on the UCF101-24 test set. All model GPU inference times are based on NVIDIA Tesla T4 machines, with precision type FP32. CPU inference speeds are based on Intel® Xeon® Gold 5117 CPU @ 2.00GHz, with 8 threads and precision type FP32.
## III. Quick Integration > ❗ Before quick integration, please install the PaddleX wheel package. For detailed instructions, refer to the [PaddleX Local Installation Guide](../../../installation/installation.en.md). After installing the wheel package, you can complete video Detection module inference with just a few lines of code. You can switch between models in this module freely, and you can also integrate the model inference of the video Detection module into your project. Before running the following code, please download the [demo video](https://paddle-model-ecology.bj.bcebos.com/paddlex/videos/demo_video/HorseRiding.avi) to your local machine. ```python from paddlex import create_model model = create_model(model_name="YOWO") output = model.predict(input="HorseRiding.avi", batch_size=1) for res in output: res.print() res.save_to_video(save_path="./output/") res.save_to_json(save_path="./output/res.json") ```[xmin, ymin, xmax, ymax]: The coordinates of the top-left and bottom-right corners of the bounding box, both are floating-point numbers.
- float: The confidence score of the bounding box, a floating-point number.
- string: The category to which the bounding box belongs, a string.
In the above Python script, the following steps are executed:
* `create_model` instantiates the video detection model (`YOWO`), with specific explanations as follows:
| Parameter | Parameter Description | Parameter Type | Options | Default Value |
|---|---|---|---|---|
model_name |
Name of the model | str |
All model names supported by PaddleX | None |
model_dir |
Path to store the model | str |
None | None |
nms_thresh |
The IoU threshold parameter in the Non-Maximum Suppression (NMS) process; if not specified, the default configuration of the PaddleX official model will be used | float/None |
Floating-point number greater than 0/None | None |
score_thresh |
The prediction confidence threshold; if not specified, the default configuration of the PaddleX official model will be used | float/None |
Floating-point number greater than 0/None | None |
| Parameter | Parameter Description | Parameter Type | Options | Default Value |
|---|---|---|---|---|
input |
Data to be predicted, supporting multiple input types | Python Var/str/list |
|
None |
batch_size |
Batch size | int |
None | 1 |
nms_thresh |
The IoU threshold parameter in the Non-Maximum Suppression (NMS) process | float|None |
|
None |
score_thresh |
Prediction confidence threshold | float|None |
|
None |
| Method | Method Description | Parameter | Parameter Type | Parameter Description | Default Value |
|---|---|---|---|---|---|
print() |
Print the result to the terminal | format_json |
bool |
Whether to format the output content using json indentation |
True |
indent |
int |
JSON formatting setting, only effective when format_json is True |
4 | ||
ensure_ascii |
bool |
JSON formatting setting, only effective when format_json is True |
False |
||
save_to_json() |
Save the result as a JSON file | save_path |
str |
The path to save the file. When it is a directory, the saved file name is consistent with the input file name | None |
indent |
int |
JSON formatting setting | 4 | ||
ensure_ascii |
bool |
JSON formatting setting | False |
||
save_to_video() |
Save the result as a video file | save_path |
str |
The path to save the file. When it is a directory, the saved file name is consistent with the input file name | None |
| Attribute | Attribute Description |
|---|---|
json |
Get the prediction result in json format |
video |
Get the visualization video and frame rate in dict format. Here, the visualization video is an np.array array with dimensions (number of video frames, video height, video width, number of video channels) |
{
"done_flag": true,
"check_pass": true,
"attributes": {
"label_file": "..\/..\/dataset\/video_det_examples\/label_map.txt",
"num_classes": 24,
"train_samples": 6878,
"train_sample_paths": [
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SoccerJuggling\/v_SoccerJuggling_g19_c06\/00296.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SkateBoarding\/v_SkateBoarding_g17_c04\/00026.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/RopeClimbing\/v_RopeClimbing_g01_c03\/00055.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/HorseRiding\/v_HorseRiding_g11_c05\/00132.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g13_c03\/00089.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Basketball\/v_Basketball_g13_c04\/00050.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g01_c05\/00024.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/RopeClimbing\/v_RopeClimbing_g03_c04\/00118.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/GolfSwing\/v_GolfSwing_g01_c06\/00231.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/TrampolineJumping\/v_TrampolineJumping_g02_c02\/00134.jpg"
],
"val_samples": 3916,
"val_sample_paths": [
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/IceDancing\/v_IceDancing_g22_c02\/00017.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/TennisSwing\/v_TennisSwing_g04_c02\/00046.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/SoccerJuggling\/v_SoccerJuggling_g08_c03\/00169.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Fencing\/v_Fencing_g24_c02\/00009.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Diving\/v_Diving_g16_c02\/00110.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/HorseRiding\/v_HorseRiding_g08_c02\/00079.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g17_c07\/00008.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/Skiing\/v_Skiing_g20_c06\/00221.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/PoleVault\/v_PoleVault_g17_c07\/00137.jpg",
"check_dataset\/..\/..\/dataset\/video_det_examples\/rgb-images\/GolfSwing\/v_GolfSwing_g24_c01\/00093.jpg"
]
},
"analysis": {
"histogram": "check_dataset\/histogram.png"
},
"dataset_path": "video_det_examples",
"show_type": "video",
"dataset_type": "VideoDetDataset"
}
The above validation results, with check_pass being True, indicate that the dataset format meets the requirements. Explanations for other indicators are as follows:
attributes.num_classes: The number of classes in this dataset is 24;attributes.train_samples: The number of training set samples in this dataset is 6878;attributes.val_samples: The number of validation set samples in this dataset is 3916;attributes.train_sample_paths: A list of relative paths to the visual samples in the training set of this dataset;attributes.val_sample_paths: A list of relative paths to the visual samples in the validation set of this dataset;Additionally, the dataset validation analyzes the sample number distribution across all classes in the dataset and generates a distribution histogram (histogram.png):
(1) Dataset Format Conversion
Image Detection does not currently support data conversion.
(2) Dataset Splitting
Image Detection does not currently support data conversion.
A single command can complete the model training. Taking the training of the video Detection model YOWO as an example:
python main.py -c paddlex/configs/modules/video_det_examples/YOWO.yaml \
-o Global.mode=train \
-o Global.dataset_dir=./dataset/video_det_examples
the following steps are required:
.yaml configuration file (here it is YOWO.yaml. When training other models, you need to specify the corresponding configuration files. The relationship between the model and configuration files can be found in the PaddleX Model List (CPU/GPU))-o Global.mode=train-o Global.dataset_dir. Other related parameters can be set by modifying the fields under Global and Train in the .yaml configuration file, or adjusted by appending parameters in the command line. For example, to specify training on the second GPU: -o Global.device=gpu:2; to set the number of training epochs to 10: -o Train.epochs_iters=10. For more modifiable parameters and their detailed explanations, refer to the configuration file parameter instructions for the corresponding task module of the model PaddleX Common Model Configuration File Parameters.output. If you need to specify a save path, you can set it through the -o Global.output field in the configuration file.After completing the model training, all outputs are saved in the specified output directory (default is ./output/), typically including:
train_result.json: Training result record file, recording whether the training task was completed normally, as well as the output weight metrics, related file paths, etc.;
train.log: Training log file, recording changes in model metrics and loss during training;config.yaml: Training configuration file, recording the hyperparameter configuration for this training session;.pdparams, .pdema, .pdopt.pdstate, .pdiparams, .pdmodel: Model weight-related files, including network parameters, optimizer, EMA, static graph network parameters, static graph network structure, etc.;After completing model training, you can evaluate the specified model weight file on the validation set to verify the model accuracy. Using PaddleX for model evaluation, a single command can complete the model evaluation:
python main.py -c paddlex/configs/modules/video_detection/YOWO.yaml \
-o Global.mode=evaluate \
-o Global.dataset_dir=./dataset/video_det_examples
Similar to model training, the following steps are required:
.yaml configuration file (here it is YOWO.yaml)-o Global.mode=evaluate-o Global.dataset_dir. Other related parameters can be set by modifying the fields under Global and Evaluate in the .yaml configuration. Other related parameters can be set by modifying the fields under Global and Evaluate in the .yaml configuration file. For details, please refer to PaddleX Common Model Configuration File Parameter Description.When evaluating the model, you need to specify the model weight file path. Each configuration file has a default weight save path built-in. If you need to change it, simply set it by appending a command line parameter, such as -o Evaluate.weight_path=./output/best_model/best_model.pdparams.
After completing the model evaluation, an evaluate_result.json file will be generated, which records the evaluation results. Specifically, it records whether the evaluation task was completed successfully and the model's evaluation metrics, including mAP;
After completing model training and evaluation, you can use the trained model weights for inference predictions or Python integration.
To perform inference prediction through the command line, simply use the following command. Before running the following code, please download the demo video to your local machine.
python main.py -c paddlex/configs/modules/video_detection/YOWO.yaml \
-o Global.mode=predict \
-o Predict.model_dir="./output/best_model/inference" \
-o Predict.input="HorseRiding.avi"
Similar to model training and evaluation, the following steps are required:
.yaml configuration file path for the model (here it is YOWO.yaml)-o Global.mode=predict-o Predict.model_dir="./output/best_model/inference"-o Predict.input="..."
Other related parameters can be set by modifying the fields under Global and Predict in the .yaml configuration file. For details, please refer to PaddleX Common Model Configuration File Parameter Description.The model can be directly integrated into the PaddleX pipelines or directly into your own project.
1.Pipeline Integration
The video Detection module can be integrated into the General Video Detection Pipeline of PaddleX. Simply replace the model path to update the video Detection module of the relevant pipeline. In pipeline integration, you can use high-performance inference and service-oriented deployment to deploy your obtained model.
2.Module Integration
The weights you produce can be directly integrated into the video Detection module. You can refer to the Python example code in Quick Integration and simply replace the model with the path to your trained model.