--- comments: true --- # Instance Segmentation Module Tutorial ## I. Overview The instance segmentation module is a crucial component in computer vision systems, responsible for identifying and marking pixels that contain specific object instances in images or videos. The performance of this module directly impacts the accuracy and efficiency of the entire computer vision system. The instance segmentation module typically outputs pixel-level masks (masks) for each target instance, which are then passed as input to the object recognition module for subsequent processing. ## II. Supported Model List > The inference time only includes the model inference time and does not include the time for pre- or post-processing.
| Model | Model Download Link | Mask AP | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Description |
|---|---|---|---|---|---|---|
| Mask-RT-DETR-H | Inference Model/Training Model | 50.6 | 180.83 / 180.83 | 1711.24 / 1711.24 | 449.9 | Mask-RT-DETR is an instance segmentation model based on RT-DETR. By adopting the high-performance PP-HGNetV2 as the backbone network and constructing a MaskHybridEncoder encoder, along with introducing IOU-aware Query Selection technology, it achieves state-of-the-art (SOTA) instance segmentation accuracy with the same inference time. |
| Mask-RT-DETR-L | Inference Model/Training Model | 45.7 | 113.20 / 113.20 | 1179.56 / 1179.56 | 113.6 |
| Model | Model Download Link | Mask AP | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Description |
|---|---|---|---|---|---|---|
| Cascade-MaskRCNN-ResNet50-FPN | Inference Model/Training Model | 36.3 | 136.79 / 136.79 | - / 5935.41 | 254.8 | Cascade-MaskRCNN is an improved Mask RCNN instance segmentation model that utilizes multiple detectors in a cascade, optimizing segmentation results by leveraging different IOU thresholds to address the mismatch between detection and inference stages, thereby enhancing instance segmentation accuracy. |
| Cascade-MaskRCNN-ResNet50-vd-SSLDv2-FPN | Inference Model/Training Model | 39.1 | 137.40 / 137.40 | - / 6816.68 | 254.7 | |
| Mask-RT-DETR-H | Inference Model/Training Model | 50.6 | 180.83 / 180.83 | 1711.24 / 1711.24 | 449.9 | Mask-RT-DETR is an instance segmentation model based on RT-DETR. By adopting the high-performance PP-HGNetV2 as the backbone network and constructing a MaskHybridEncoder encoder, along with introducing IOU-aware Query Selection technology, it achieves state-of-the-art (SOTA) instance segmentation accuracy with the same inference time. |
| Mask-RT-DETR-L | Inference Model/Training Model | 45.7 | 113.20 / 113.20 | 1179.56 / 1179.56 | 113.6 | |
| Mask-RT-DETR-M | Inference Model/Training Model | 42.7 | 87.08 / 87.08 | - / 2090.73 | 66.6 | |
| Mask-RT-DETR-S | Inference Model/Training Model | 41.0 | 120.86 / 120.86 | - / 2163.07 | 51.8 | |
| Mask-RT-DETR-X | Inference Model/Training Model | 47.5 | 141.43 / 141.43 | 1379.14 / 1379.14 | 237.5 | |
| MaskRCNN-ResNet50-FPN | Inference Model/Training Model | 35.6 | 112.79 / 112.79 | - / 4912.37 | 157.5 | Mask R-CNN is a full-task deep learning model from Facebook AI Research (FAIR) that can perform object classification and localization in a single model, combined with image-level masks to complete segmentation tasks. |
| MaskRCNN-ResNet50-vd-FPN | Inference Model/Training Model | 36.4 | 112.88 / 112.88 | - / 5204.97 | 157.5 | |
| MaskRCNN-ResNet50 | Inference Model/Training Model | 32.8 | 181.60 / 181.60 | - / 5523.45 | 128.7 | |
| MaskRCNN-ResNet101-FPN | Inference Model/Training Model | 36.6 | 138.84 / 138.84 | - / 5107.74 | 225.4 | |
| MaskRCNN-ResNet101-vd-FPN | Inference Model/Training Model | 38.1 | 141.73 / 141.73 | - / 5592.76 | 225.1 | |
| MaskRCNN-ResNeXt101-vd-FPN | Inference Model/Training Model | 39.5 | 220.83 / 220.83 | - / 5932.59 | 370.0 | |
| PP-YOLOE_seg-S | Inference Model/Training Model | 32.5 | 243.41 / 222.30 | 2507.70 / 1282.35 | 31.5 | PP-YOLOE_seg is an instance segmentation model based on PP-YOLOE. This model inherits PP-YOLOE's backbone and head, significantly enhancing instance segmentation performance and inference speed through the design of a PP-YOLOE instance segmentation head. |
| SOLOv2 | Inference Model/Training Model | 35.5 | 131.99 / 131.99 | - / 2369.98 | 179.1 | SOLOv2 is a real-time instance segmentation algorithm that segments objects by location. This model is an improved version of SOLO, achieving a good balance between accuracy and speed through the introduction of mask learning and mask NMS. |
| Mode | GPU Configuration | CPU Configuration | Acceleration Technology Combination |
|---|---|---|---|
| Normal Mode | FP32 Precision / No TRT Acceleration | FP32 Precision / 8 Threads | PaddleInference |
| High-Performance Mode | Optimal combination of pre-selected precision types and acceleration strategies | FP32 Precision / 8 Threads | Pre-selected optimal backend (Paddle/OpenVINO/TRT, etc.) |
[xmin, ymin, xmax, ymax].
- `pred`: The actual mask predicted by the instance segmentation model. Since the data is too large to be printed directly, it is replaced with `...` here. You can use `res.save_to_img()` to save the prediction results as an image and `res.save_to_json()` to save the prediction results as a JSON file.
The visualization image is as follows:
**Note:** Due to network issues, the above URL may not be accessible. If you need to access this link, please check the validity of the URL and try again. If the problem persists, it may be related to the link itself or the network connection.
Related methods, parameters, and explanations are as follows:
* `create_model` instantiates a general instance segmentation model (here, `PP-YOLOE_seg-S` is used as an example), and the specific explanations are as follows:
| Parameter | Parameter Description | Parameter Type | Options | Default Value |
|---|---|---|---|---|
model_name |
Name of the model | str |
None | None |
model_dir |
Path to store the model | str |
None | None |
device |
The device used for model inference | str |
It supports specifying specific GPU card numbers, such as "gpu:0", other hardware card numbers, such as "npu:0", or CPU, such as "cpu". | gpu:0 |
threshold |
Threshold for filtering low-confidence objects | float/None |
None | None |
use_hpip |
Whether to enable the high-performance inference plugin | bool |
None | False |
hpi_config |
High-performance inference configuration | dict | None |
None | None |
| Parameter | Parameter Description | Parameter Type | Options | Default Value |
|---|---|---|---|---|
input |
Data to be predicted, supporting multiple input types | Python Var/str/list |
|
None |
batch_size |
Batch size | int |
Any integer | 1 |
threshold |
Threshold for filtering low-confidence objects | float/None |
|
None |
| Method | Method Description | Parameter | Parameter Type | Parameter Description | Default Value |
|---|---|---|---|---|---|
print() |
Print the results to the terminal | format_json |
bool |
Whether to format the output content using JSON indentation |
True |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable, only effective when format_json is True |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. If set to True, all non-ASCII characters will be escaped; False retains the original characters, only effective when format_json is True |
False
| ||
save_to_json() |
Save the results as a JSON file | save_path |
str |
The path to save the file. If it is a directory, the saved file name will be consistent with the input file name | None |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable, only effective when format_json is True |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. If set to True, all non-ASCII characters will be escaped; False retains the original characters, only effective when format_json is True |
False
| ||
save_to_img() |
Save the results as an image file | save_path |
str |
The path to save the file. If it is a directory, the saved file name will be consistent with the input file name | None |
| Attribute | Attribute Description |
|---|---|
json |
Get the prediction result in json format |
img |
Get the visualization image in dict format |
The specific content of the validation result file is:
{
"done_flag": true,
"check_pass": true,
"attributes": {
"num_classes": 2,
"train_samples": 79,
"train_sample_paths": [
"check_dataset/demo_img/pexels-photo-634007.jpeg",
"check_dataset/demo_img/pexels-photo-59576.png"
],
"val_samples": 19,
"val_sample_paths": [
"check_dataset/demo_img/peasant-farmer-farmer-romania-botiza-47862.jpeg",
"check_dataset/demo_img/pexels-photo-715546.png"
]
},
"analysis": {
"histogram": "check_dataset/histogram.png"
},
"dataset_path": "instance_seg_coco_examples",
"show_type": "image",
"dataset_type": "COCOInstSegDataset"
}
In the above verification results, check_pass being True indicates that the dataset format meets the requirements. Explanations for other indicators are as follows:
attributes.num_classes: The number of classes in this dataset is 2;attributes.train_samples: The number of training samples in this dataset is 79;attributes.val_samples: The number of validation samples in this dataset is 19;attributes.train_sample_paths: A list of relative paths to the visualized training samples in this dataset;attributes.val_sample_paths: A list of relative paths to the visualized validation samples in this dataset;
Additionally, the dataset verification also analyzes the distribution of sample numbers across all categories in the dataset and generates a distribution histogram (histogram.png):
(1) Dataset Format Conversion
The instance segmentation task supports converting LabelMe format to COCO format. The parameters for dataset format conversion can be set by modifying the fields under CheckDataset in the configuration file. Below are some example explanations for some of the parameters in the configuration file:
CheckDataset:convert:enable: Whether to perform dataset format conversion. Set to True to enable dataset format conversion, default is False;src_dataset_type: If dataset format conversion is performed, the source dataset format needs to be set. The available source format is LabelMe;
For example, if you want to convert a LabelMe dataset to COCO format, you need to modify the configuration file as follows:cd /path/to/paddlex
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/data/instance_seg_labelme_examples.tar -P ./dataset
tar -xf ./dataset/instance_seg_labelme_examples.tar -C ./dataset/
......
CheckDataset:
......
convert:
enable: True
src_dataset_type: LabelMe
......
Then execute the command:
python main.py -c paddlex/configs/modules/instance_segmentation/Mask-RT-DETR-L.yaml\
-o Global.mode=check_dataset \
-o Global.dataset_dir=./dataset/instance_seg_labelme_examples
After the data conversion is executed, the original annotation files will be renamed to xxx.bak in the original path.
The above parameters also support being set by appending command line arguments:
python main.py -c paddlex/configs/modules/instance_segmentation/Mask-RT-DETR-L.yaml\
-o Global.mode=check_dataset \
-o Global.dataset_dir=./dataset/instance_seg_labelme_examples \
-o CheckDataset.convert.enable=True \
-o CheckDataset.convert.src_dataset_type=LabelMe
(2) Dataset Splitting
The parameters for dataset splitting can be set by modifying the fields under CheckDataset in the configuration file. Some example explanations for the parameters in the configuration file are as follows:
CheckDataset:split:enable: Whether to re-split the dataset. When set to True, the dataset will be re-split. The default is False;train_percent: If the dataset is to be re-split, the percentage of the training set needs to be set. The type is any integer between 0-100, and the sum with val_percent must be 100;For example, if you want to re-split the dataset with a 90% training set and a 10% validation set, you need to modify the configuration file as follows:
......
CheckDataset:
......
split:
enable: True
train_percent: 90
val_percent: 10
......
Then execute the command:
python main.py -c paddlex/configs/modules/instance_segmentation/Mask-RT-DETR-L.yaml \
-o Global.mode=check_dataset \
-o Global.dataset_dir=./dataset/instance_seg_labelme_examples
After data splitting, the original annotation files will be renamed as xxx.bak in the original path.
The above parameters can also be set by appending command line arguments:
python main.py -c paddlex/configs/modules/instance_segmentation/Mask-RT-DETR-L.yaml \
-o Global.mode=check_dataset \
-o Global.dataset_dir=./dataset/instance_seg_labelme_examples \
-o CheckDataset.split.enable=True \
-o CheckDataset.split.train_percent=90 \
-o CheckDataset.split.val_percent=10
output. If you need to specify a save path, you can set it through the -o Global.output field in the configuration file.After completing the model training, all outputs are saved in the specified output directory (default is ./output/), typically including:
train_result.json: Training result record file, recording whether the training task was completed normally, as well as the output weight metrics, related file paths, etc.;
train.log: Training log file, recording changes in model metrics and loss during training;config.yaml: Training configuration file, recording the hyperparameter configuration for this training session;.pdparams, .pdema, .pdopt.pdstate, .pdiparams, .json: Model weight-related files, including network parameters, optimizer, EMA, static graph network parameters, static graph network structure, etc.;.json file) from protobuf(the former.pdmodel file) to be compatible with PIR and more flexible and scalable.When evaluating the model, you need to specify the model weights file path. Each configuration file has a default weight save path built-in. If you need to change it, simply set it by appending a command line parameter, such as -o Evaluate.weight_path=./output/best_model/best_model.pdparams.
After completing the model evaluation, an evaluate_result.json file will be generated, which records the evaluation results, specifically whether the evaluation task was completed successfully and the model's evaluation metrics, including AP.