polish code & doc

docs/tutorials/INSTALL.md

@@ -22,7 +22,7 @@ sudo docker run --name paddlex -v $PWD:/paddle --shm-size=8G --network=host -it
 
 **注意**：
 * 首次使用该镜像时，下述命令会自动下载该镜像文件，下载需要一定的时间，请耐心等待；
-* 请使用 **2.5.2** 及更高版本的 PaddlePaddle；
+* 请使用 **3.0** 版本的 PaddlePaddle；
 * 上述命令会创建一个名为 paddlex 的 Docker 容器，之后再次使用该容器时无需再次运行该命令；
 * 参数 `--shm-size=8G` 将设置容器的共享内存为 8G，如机器环境允许，建议将该参数设置较大，如 `64G`；
 
@@ -42,10 +42,7 @@ python -m pip install paddlepaddle==2.6.1 -f https://www.paddlepaddle.org.cn/whl
 更多飞桨 Wheel 版本请参考[飞桨官网](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/pip/linux-pip.html)。
 
 #### 更多安装方式
-
-<!-- 这里需要指定内多硬件安装的文档 -->
-关于**源码编译**安装等更多安装方式，及**昆仑芯**、**海光**、**寒武纪**、**昇腾**等飞桨版本，请参考[飞桨官网](https://www.paddlepaddle.org.cn/install/quick?docurl=undefined)。
-
+关于其他硬件安装飞桨，请参考[多硬件安装飞桨](./INSTALL_OTHER_DEVICES.md)。
 
 ### 1.2 验证
 

docs/tutorials/INSTALL_OTHER_DEVICES.md

@@ -0,0 +1,116 @@
+# 多硬件安装飞桨
+本文档主要针对昇腾 NPU、寒武纪 MLU、昆仑 XPU 硬件平台，介绍如何安装飞桨。
+## 1. 昇腾 NPU 飞桨安装
+### 1.1 环境准备
+当前 PaddleX 支持昇腾 910B 芯片，昇腾驱动版本为 23.0.3。考虑到环境差异性，我们推荐使用飞桨官方提供的标准镜像完成环境准备。
+- 1.拉取镜像，此镜像仅为开发环境，镜像中不包含预编译的飞桨安装包，镜像中已经默认安装了昇腾算子库 CANN-8.0.RC1。
+
+```
+docker pull registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-aarch64-gcc84-py39 # 适用于 ARM 架构
+docker pull registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-x86_64-gcc84-py39 # 适用于 X86 架构
+```
+
+- 2.参考如下命令启动容器，ASCEND_RT_VISIBLE_DEVICES 指定可见的 NPU 卡号
+```
+docker run -it --name paddle-npu-dev -v $(pwd):/work \
+    --privileged --network=host --shm-size=128G -w=/work \
+    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
+    -v /usr/local/dcmi:/usr/local/dcmi \
+    -e ASCEND_RT_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" \
+    registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-$(uname -m)-gcc84-py39 /bin/bash
+```
+### 1.2 安装paddle包
+当前提供 Python3.9 的 wheel 安装包。如有其他 Python 版本需求，可以参考[飞桨官方文档](https://www.paddlepaddle.org.cn/install/quick)自行编译安装。
+
+- 1.下载安装 Python3.9 的 wheel 安装包
+
+```
+# 注意需要先安装飞桨CPU版本
+pip install 
+pip install
+```
+- 2.验证安装包
+安装完成之后，运行如下命令。
+```
+python -c "import paddle; paddle.utils.run_check()"
+```
+预期得到如下输出结果
+```
+Running verify PaddlePaddle program ...
+PaddlePaddle works well on 1 npu.
+PaddlePaddle works well on 8 npus.
+PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
+```
+
+## 2. 寒武纪 MLU 飞桨安装
+### 2.1 环境准备
+考虑到环境差异性，我们推荐使用飞桨官方提供的标准镜像完成环境准备。
+- 拉取镜像，此镜像仅为开发环境，镜像中不包含预编译的飞桨安装包
+```
+# 适用于 X86 架构，暂时不提供 Arrch64 架构镜像
+docker pull registry.baidubce.com/device/paddle-mlu:ctr2.15.0-ubuntu20-gcc84-py310
+```
+- 参考如下命令启动容器
+```
+docker run -it --name paddle-mlu-dev -v $(pwd):/work \
+  -w=/work --shm-size=128G --network=host --privileged  \
+  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
+  -v /usr/bin/cnmon:/usr/bin/cnmon \
+  registry.baidubce.com/device/paddle-mlu:ctr2.15.0-ubuntu20-gcc84-py310 /bin/bash
+```
+### 2.2 安装paddle包
+当前提供 Python3.10 的 wheel 安装包。有其他 Python 版本需求，可以参考[飞桨官方文档](https://www.paddlepaddle.org.cn/install/quick)自行编译安装。
+
+- 1.下载安装 Python3.10 的wheel 安装包。
+```
+# 注意需要先安装飞桨 CPU 版本
+python -m pip install paddlepaddle -i https://www.paddlepaddle.org.cn/packages/nightly/cpu/
+python -m pip install --pre paddle-custom-mlu -i https://www.paddlepaddle.org.cn/packages/nightly/mlu/
+```
+- 2.验证安装包
+安装完成之后，运行如下命令。
+```
+python -c "import paddle; paddle.utils.run_check()"
+```
+预期得到如下输出结果
+```
+Running verify PaddlePaddle program ...
+PaddlePaddle works well on 1 mlu.
+PaddlePaddle works well on 16 mlus.
+PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
+```
+
+# 3.昆仑 XPU 飞桨安装
+## 3.1 环境准备
+考虑到环境差异性，我们推荐使用飞桨官方发布的昆仑 XPU 开发镜像，该镜像预装有昆仑基础运行环境库（XRE）。
+- 1.拉取镜像，此镜像仅为开发环境，镜像中不包含预编译的飞桨安装包
+```
+docker pull registry.baidubce.com/device/paddle-xpu:ubuntu20-x86_64-gcc84-py310 # X86 架构
+docker pull registry.baidubce.com/device/paddle-xpu:kylinv10-aarch64-gcc82-py310 # ARM 架构
+```
+- 2.参考如下命令启动容器
+```
+docker run -it --name=xxx -m 81920M --memory-swap=81920M \
+    --shm-size=128G --privileged --net=host \
+    -v $(pwd):/workspace -w /workspace \
+    registry.baidubce.com/device/paddle-xpu:kylinv10-aarch64-gcc82 bash
+```
+
+## 3.2 安装paddle包
+当前提供 Python3.10 的 wheel 安装包。有其他 Python 版本需求，可以参考[飞桨官方文档](https://www.paddlepaddle.org.cn/install/quick)自行编译安装。
+
+- 1.安装 Python3.10 的 wheel 安装包
+```
+pip install https://paddle-wheel.bj.bcebos.com/2.6.1/xpu/paddlepaddle_xpu-2.6.1-cp310-cp310-linux_x86_64.whl # X86 架构
+pip install https://paddle-device.bj.bcebos.com/2.6.1/xpu/paddlepaddle_xpu-2.6.1-cp310-cp310-linux_aarch64.whl # ARM 架构
+```
+- 2.验证安装包
+安装完成之后，运行如下命令
+```
+python -c "import paddle; paddle.utils.run_check()"
+```
+预期得到如下输出结果
+```
+PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
+```

docs/tutorials/data/annotation/OCRAnnoTools.md

@@ -0,0 +1,58 @@
+# 数据标注指南
+本文档将介绍如何使用 [PPOCRLabel](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.7/PPOCRLabel/README_ch.md) 完成 PP-OCR 单模型和表格识别的数据标注。 
+
+点击上述链接，参考首页文档即可安装数据标注工具并查看详细使用流程，以下提供简洁版本说明：
+
+## 1.安装paddlepaddle
+
+安装过程请参考[飞桨安装文档]()
+## 2.安装与运行 PPOCRLabel
+
+PPOCRLabel 可通过 wheel 包与 Python 脚本两种方式启动，wheel 包形式启动更加方便，这里只提供 whl 包说明：
+
+* windows 安装：
+
+```
+pip install PPOCRLabel  # 安装
+
+# 选择标签模式来启动
+PPOCRLabel --lang ch  # 启动【普通模式】，用于打【检测+识别】场景的标签
+```
+
+* MacOS
+
+```
+pip3 install PPOCRLabel
+pip3 install opencv-contrib-python-headless==4.2.0.32 # 如果下载过慢请添加"-i https://mirror.baidu.com/pypi/simple"
+
+# 选择标签模式来启动
+PPOCRLabel --lang ch  # 启动【普通模式】，用于打【检测+识别】场景的标签
+```
+
+**注：** 更多环境安装请参考[详细说明](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.7/PPOCRLabel/README_ch.md)。
+
+## 3.操作步骤
+
+**文本检测和文本识别的标注过程：**
+
+1. 安装与运行：使用上述命令安装与运行程序。
+2. 打开文件夹：在菜单栏点击 “文件” - "打开目录" 选择待标记图片的文件夹.
+3. 自动标注：点击 ”自动标注“，使用 PP-OCR 超轻量模型对图片文件名前图片状态为 “X” 的图片进行自动标注。
+4. 手动标注：点击 “矩形标注”（推荐直接在英文模式下点击键盘中的 “W”)，用户可对当前图片中模型未检出的部分进行手动绘制标记框。点击键盘 Q，则使用四点标注模式（或点击“编辑” - “四点标注”），用户依次点击 4 个点后，双击左键表示标注完成。
+5. 标记框绘制完成后，用户点击 “确认”，检测框会先被预分配一个 “待识别” 标签。
+6. 重新识别：将图片中的所有检测画绘制/调整完成后，点击 “重新识别”，PP-OCR 模型会对当前图片中的 **所有检测框** 重新识别。
+7. 内容更改：单击识别结果，对不准确的识别结果进行手动更改。
+8. **确认标记：点击 “确认”，图片状态切换为 “√”，跳转至下一张。**
+9. 删除：点击 “删除图像”，图片将会被删除至回收站。
+10. 导出结果：用户可以通过菜单中“文件-导出标记结果”手动导出，同时也可以点击“文件 - 自动导出标记结果”开启自动导出。手动确认过的标记将会被存放在所打开图片文件夹下的 Label.txt 中。在菜单栏点击 “文件” - "导出识别结果"后，会将此类图片的识别训练数据保存在 crop_img 文件夹下，识别标签保存在 rec_gt.txt 中。
+
+**表格识别的标注过程：**
+
+表格标注针对表格的结构化提取，将图片中的表格转换为 Excel 格式，因此标注时需要配合外部软件打开 Excel 同时完成。在 PPOCRLabel 软件中完成表格中的文字信息标注（文字与位置）、在 Excel 文件中完成表格结构信息标注，推荐的步骤为：
+1. 表格识别：打开表格图片后，点击软件右上角表格识别按钮，软件调用 PP-Structure 中的表格识别模型，自动为表格打标签，同时弹出 Excel。
+2. 更改标注结果：**以表格中的单元格为单位增加标注框**（即一个单元格内的文字都标记为一个框）。标注框上鼠标右键后点击单元格重识别可利用模型自动识别单元格内的文字。
+3. **调整单元格顺序：** 点击软件视图-显示框编号打开标注框序号，在软件界面右侧拖动识别结果一栏下的所有结果，使得标注框编号按照从左到右，从上到下的顺序排列，按行依次标注。
+4. 标注表格结构：**在外部 Excel 软件中，将存在文字的单元格标记为任意标识符（如1）** ，保证 Excel 中的单元格合并情况与原图相同即可（即不需要 Excel 中的单元格文字与图片中的文字完全相同）。
+5. 导出 JSON 格式：关闭所有表格图像对应的Excel，点击文件-导出表格标注，生成 gt.txt 标注文件。
+
+遇到问题可以参考：[操作视频](https://www.bilibili.com/video/BV1wR4y1v7JE/?share_source=copy_web&vd_source=cf1f9d24648d49636e3d109c9f9a377d&t=1998)

docs/tutorials/data/annotation/README.md

@@ -1,9 +1,13 @@
 # 数据标注
 
-| 任务类型  |     数据标注及准备文档   |
+如您使用 PaddleX 进行模型训练，您需要根据任务模块类型选择对应的数据标注方式准备数据集。
+
+| 任务模块类型  |     数据标注及准备文档   |
 |:--------:| :--------:|
 |    图像分类   |   [文档链接](ClsAnnoTools.md)  |
 |    目标检测   |   [文档链接](DetAnnoTools.md)  |
 |    语义分割   |   [文档链接](SegAnnoTools.md)  |
 |    实例分割   |   [文档链接](InsSegAnnoTools.md)  |
+|    OCR/表格识别   |   [文档链接](OCRAnnoTools.md)  |
+
 

docs/tutorials/data/dataset_check.md

@@ -24,9 +24,43 @@ python main.py -c paddlex/configs/image_classification/PP-LCNet_x1_0.yaml \
     -o Global.dataset_dir=./dataset/cls_flowers_examples
 ```
 
-执行上述命令后，PaddleX 会对数据集进行校验，并统计数据集的基本信息。相关信息会保存在当前目录的 `./output/check_dataset` 目录下。
+执行上述命令后，PaddleX 会对数据集进行校验，并统计数据集的基本信息。命令运行成功后会在log中打印出 `Check dataset passed !` 信息，同时相关产出会保存在当前目录的 `./output/check_dataset` 目录下，产出目录中包括可视化的示例样本图片和样本分布直方图。校验结果文件保存在 `./output/check_dataset_result.json`，校验结果文件具体内容为
+```
+{
+  "done_flag": true,
+  "check_pass": true,
+  "attributes": {
+    "label_file": "dataset/label.txt",
+    "num_classes": 102,
+    "train_samples": 1020,
+    "train_sample_paths": [
+      "check_dataset/demo_img/image_01904.jpg",
+      "check_dataset/demo_img/image_06940.jpg"
+    ],
+    "val_samples": 1020,
+    "val_sample_paths": [
+      "check_dataset/demo_img/image_01937.jpg",
+      "check_dataset/demo_img/image_06958.jpg"
+    ]
+  },
+  "analysis": {
+    "histogram": "check_dataset/histogram.png"
+  },
+  "dataset_path": "./dataset/cls_flowers_examples",
+  "show_type": "image",
+  "dataset_type": "ClsDataset"
+}
+```
+上述校验结果中，check_pass 为 True 表示数据集格式符合要求，其他部分指标的说明如下：
+
+- attributes.num_classes：该数据集类别数为 102；
+- attributes.train_samples：该数据集训练集样本数量为 1020；
+- attributes.val_samples：该数据集验证集样本数量为 1020；
+- attributes.train_sample_paths：该数据集训练集样本可视化图片相对路径列表；
+- attributes.val_sample_paths：该数据集验证集样本可视化图片相对路径列表；
 
-<!-- 这里需要增加输出的说明，以及产出的说明，畅达 -->
+另外，数据集校验还对数据集中所有类别的样本数量分布情况进行了分析，并绘制了分布直方图（histogram.png）：
+![样本分布直方图](https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/doc_images/open_source/quick_start/histogram.png)
 
 **注**：只有通过数据校验的数据才可以训练和评估。
 
@@ -38,23 +72,174 @@ python main.py -c paddlex/configs/image_classification/PP-LCNet_x1_0.yaml \
 数据集校验相关的参数可以通过修改配置文件中 `CheckDataset` 下的字段进行设置，配置文件中部分参数的示例说明如下：
 
 * `CheckDataset`:
-    <!-- 这里需要增加详细的说明，廷权 -->
-    * `dst_dataset_name`: 生成的数据集目录名，PaddleX 在数据校验时，会产生一个新的数据集；
+    * `convert`:
+        * `enable`: 是否进行数据集格式转换，为 `True` 时进行数据集格式转换，默认为 `False`;
+        * `src_dataset_type`: 如果进行数据集格式转换，则需设置源数据集格式；
+    * `split`:
+        * `enable`: 是否进行重新划分数据集，为 `True` 时进行数据集格式转换，默认为 `False`；
+        * `train_percent`: 如果重新划分数据集，则需要设置训练集的百分比，类型为0-1之间的任意小数，需要保证和 `val_percent` 值加和为1；
+        * `val_percent`: 如果重新划分数据集，则需要设置验证集的百分比，类型为0-1之间的任意小数，需要保证和 `train_percent` 值加和为1；
+
+数据转换和数据划分支持同时开启，对于数据划分原有标注文件会被在原路径下重命名为 `xxx.bak`，以上参数同样支持通过追加命令行参数的方式进行设置，例如如重新划分数据集并设置训练集与验证集比例：`-o CheckDataset.split=True -o CheckDataset.train_percent=0.8 -o CheckDataset.val_percent=0.2`。
+
+
+## 2.目标检测任务模块数据校验
+
+### 2.1 数据准备
+
+您需要按照 PaddleX 支持的数据格式要求准备数据，关于数据标注，您可以参考[PaddleX 数据标注](./annotation/README.md)，关于数据格式介绍，您可以参考[PaddleX 数据格式介绍](./dataset_format.md)，此处我们准备了目标检测 Demo 数据供您使用。
+
+```bash
+cd /path/to/paddlex
+wget https://paddle-model-ecology.bj.bcebos.com/paddlex/data/det_coco_examples.tar -P ./dataset
+tar -xf ./dataset/det_coco_examples.tar -C ./dataset/
+```
+
+### 2.2 数据集校验
+
+在对数据集校验时，只需一行命令：
+
+```bash
+python main.py -c paddlex/configs/object_detection/PicoDet-S.yaml \
+    -o Global.mode=check_dataset \
+    -o Global.dataset_dir=./dataset/det_coco_examples
+```
+
+执行上述命令后，PaddleX 会对数据集进行校验，并统计数据集的基本信息。命令运行成功后会在log中打印出 `Check dataset passed !` 信息，同时相关产出会保存在当前目录的 `./output/check_dataset` 目录下，产出目录中包括可视化的示例样本图片和样本分布直方图。校验结果文件保存在 `./output/check_dataset_result.json`，校验结果文件具体内容为
+```
+{
+  "done_flag": true,
+  "check_pass": true,
+  "attributes": {
+    "num_classes": 3,
+    "train_samples": 56,
+    "train_sample_paths": [
+      "check_dataset/demo_img/304.png",
+      "check_dataset/demo_img/322.png"
+    ],
+    "val_samples": 14,
+    "val_sample_paths": [
+      "check_dataset/demo_img/114.png",
+      "check_dataset/demo_img/206.png"
+    ]
+  },
+  "analysis": {
+    "histogram": "check_dataset/histogram.png"
+  },
+  "dataset_path": "./dataset/det_coco_examples",
+  "show_type": "image",
+  "dataset_type": "COCODetDataset"
+}  
+```
+上述校验结果中，check_pass 为 True 表示数据集格式符合要求，其他部分指标的说明如下：
+
+- attributes.num_classes：该数据集类别数为 3；
+- attributes.train_samples：该数据集训练集样本数量为 56；
+- attributes.val_samples：该数据集验证集样本数量为 14；
+- attributes.train_sample_paths：该数据集训练集样本可视化图片相对路径列表；
+- attributes.val_sample_paths：该数据集验证集样本可视化图片相对路径列表；
+
+另外，数据集校验还对数据集中所有类别的样本数量分布情况进行了分析，并绘制了分布直方图（histogram.png）：
+![样本分布直方图](https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/doc_images/open_source/tutorials/data/dataset_check/object_detection/histogram.png)
+
+**注**：只有通过数据校验的数据才可以训练和评估。
+
+
+### 2.3 数据集格式转换/数据集划分（非必选）
+
+如需对数据集格式进行转换或是重新划分数据集，可通过修改配置文件或是追加超参数的方式进行设置。
 
+数据集校验相关的参数可以通过修改配置文件中 `CheckDataset` 下的字段进行设置，配置文件中部分参数的示例说明如下：
+
+* `CheckDataset`:
     * `convert`:
-        * `enable`: 是否进行数据集格式转换；
+        * `enable`: 是否进行数据集格式转换，为 `True` 时进行数据集格式转换，默认为 `False`;
         * `src_dataset_type`: 如果进行数据集格式转换，则需设置源数据集格式；
     * `split`:
-        * `enable`: 是否进行重新划分数据集；
-        * `train_percent`: 如果重新划分数据集，则需要设置训练集的百分比；
-        * `val_percent`: 如果重新划分数据集，则需要设置验证集的百分比；
+        * `enable`: 是否进行重新划分数据集，为 `True` 时进行数据集格式转换，默认为 `False`；
+        * `train_percent`: 如果重新划分数据集，则需要设置训练集的百分比，类型为0-1之间的任意小数，需要保证和 `val_percent` 值加和为1；
+        * `val_percent`: 如果重新划分数据集，则需要设置验证集的百分比，类型为0-1之间的任意小数，需要保证和 `train_percent` 值加和为1；
+
+数据转换和数据划分支持同时开启，对于数据划分原有标注文件会被在原路径下重命名为 `xxx.bak`，以上参数同样支持通过追加命令行参数的方式进行设置，例如如重新划分数据集并设置训练集与验证集比例：`-o CheckDataset.split=True -o CheckDataset.train_percent=0.8 -o CheckDataset.val_percent=0.2`。
+
+## 3.语义分割任务模块数据校验
 
-以上参数同样支持通过追加命令行参数的方式进行设置，如重新划分数据集并设置训练集与验证集比例：`-o CheckDataset.split=True -o CheckDataset.train_percent=0.8 -o CheckDataset.val_percent=0.2`。
+### 3.1 数据准备
 
+您需要按照 PaddleX 支持的数据格式要求准备数据，关于数据标注，您可以参考[PaddleX 数据标注](./annotation/README.md)，关于数据格式介绍，您可以参考[PaddleX 数据格式介绍](./dataset_format.md)，此处我们准备了语义分割 Demo 数据供您使用。
+
+```bash
+cd /path/to/paddlex
+wget https://paddle-model-ecology.bj.bcebos.com/paddlex/data/seg_optic_examples.tar -P ./dataset
+tar -xf ./dataset/seg_optic_examples.tar -C ./dataset/
+```
 
-## 目标检测任务模块数据校验
+### 3.2 数据集校验
+
+在对数据集校验时，只需一行命令：
+
+```bash
+python main.py -c paddlex/configs/semantic_segmentation/PP-LiteSeg-T.yaml \
+    -o Global.mode=check_dataset \
+    -o Global.dataset_dir=./dataset/seg_optic_examples
+```
+
+执行上述命令后，PaddleX 会对数据集进行校验，并统计数据集的基本信息。命令运行成功后会在log中打印出 `Check dataset passed !` 信息，同时相关产出会保存在当前目录的 `./output/check_dataset` 目录下，产出目录中包括可视化的示例样本图片和样本分布直方图。校验结果文件保存在 `./output/check_dataset_result.json`，校验结果文件具体内容为
+```
+{
+  "done_flag": true,
+  "check_pass": true,
+  "attributes": {
+    "train_sample_paths": [
+      "check_dataset/demo_img/P0005.jpg",
+      "check_dataset/demo_img/P0050.jpg"
+    ],
+    "train_samples": 267,
+    "val_sample_paths": [
+      "check_dataset/demo_img/N0139.jpg",
+      "check_dataset/demo_img/P0137.jpg"
+    ],
+    "val_samples": 76,
+    "num_classes": 2
+  },
+  "analysis": {
+    "histogram": "check_dataset/histogram.png"
+  },
+  "dataset_path": "./dataset/seg_optic_examples",
+  "show_type": "image",
+  "dataset_type": "SegDataset"
+}
+```
+上述校验结果中，check_pass 为 True 表示数据集格式符合要求，其他部分指标的说明如下：
+
+- attributes.num_classes：该数据集类别数为 2；
+- attributes.train_samples：该数据集训练集样本数量为 267；
+- attributes.val_samples：该数据集验证集样本数量为 76；
+- attributes.train_sample_paths：该数据集训练集样本可视化图片相对路径列表；
+- attributes.val_sample_paths：该数据集验证集样本可视化图片相对路径列表；
+
+另外，数据集校验还对数据集中所有类别的样本数量分布情况进行了分析，并绘制了分布直方图（histogram.png）：
+![样本分布直方图](https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/doc_images/open_source/tutorials/data/dataset_check/semantic_segmentation/histogram.png)
+
+**注**：只有通过数据校验的数据才可以训练和评估。
+
+
+### 3.3 数据集格式转换/数据集划分（非必选）
+
+如需对数据集格式进行转换或是重新划分数据集，可通过修改配置文件或是追加超参数的方式进行设置。
+
+数据集校验相关的参数可以通过修改配置文件中 `CheckDataset` 下的字段进行设置，配置文件中部分参数的示例说明如下：
+
+* `CheckDataset`:
+    * `convert`:
+        * `enable`: 是否进行数据集格式转换，为 `True` 时进行数据集格式转换，默认为 `False`;
+        * `src_dataset_type`: 如果进行数据集格式转换，则需设置源数据集格式；
+    * `split`:
+        * `enable`: 是否进行重新划分数据集，为 `True` 时进行数据集格式转换，默认为 `False`；
+        * `train_percent`: 如果重新划分数据集，则需要设置训练集的百分比，类型为0-1之间的任意小数，需要保证和 `val_percent` 值加和为1；
+        * `val_percent`: 如果重新划分数据集，则需要设置验证集的百分比，类型为0-1之间的任意小数，需要保证和 `train_percent` 值加和为1；
 
-## 语义分割任务模块数据校验
+数据转换和数据划分支持同时开启，对于数据划分原有标注文件会被在原路径下重命名为 `xxx.bak`，以上参数同样支持通过追加命令行参数的方式进行设置，例如如重新划分数据集并设置训练集与验证集比例：`-o CheckDataset.split=True -o CheckDataset.train_percent=0.8 -o CheckDataset.val_percent=0.2`。
 
 ## 实例分割任务模块数据校验
 

docs/tutorials/pipeline.md

@@ -1,8 +1,29 @@
-# PaddleX 产线（Pipeline）推理
+# PaddleX 模型产线开发工具
 
-PaddleX 中提供了多个产线，包括：OCR、图像分类、目标检测、实例分割、语义分割等，每个产线有多个模型可供选择，并均提供了官方预训练权重，支持通过 Python API、命令行方式直接推理预测。各产线使用方式可参考以下代码。
+PaddleX 中提供了多个模型产线，包括：OCR、图像分类、目标检测、实例分割、语义分割等，每个模型产线有多个模型可供选择，并均提供了官方权重，支持通过命令行方式直接推理预测和 Python API 预测。命令行方式直接推理预测可以快速体验模型推理效果，而 Python API 预测可以方便地集成到自己的项目中进行预测。
 
-## OCR
+## 1.安装 PaddleX
+在使用模型产线工具之前，首先需要安装 PaddleX，安装方式请参考 [PaddleX 安装文档](xxx)。
+
+
+## 2.PaddleX 模型产线工具使用方式
+### 2.1 OCR 产线
+OCR 产线内置了 PP-OCRv4 模型，包括文字检测和文字识别两个部分。文字检测支持的模型有`PP-OCRv4_mobile_det`、`PP-OCRv4_server_det`，文字识别支持的模型有`PP-OCRv4_mobile_rec`、`PP-OCRv4_server_rec`。您可以使用以下两种方式进行推理预测，如果在您的场景中，上述模型不能满足您的需求，您可以参考 [PaddleX 模型训练文档](./train/README.md) 进行训练，训练后的模型可以非常方便地集成到该产线中。
+
+<details>
+<summary><b> 命令行使用方式 </b></summary>
+您可以使用命令行将图片的文字识别出来，命令行使用方式如下：
+
+```
+paddlex --task ocrdet --model PP-OCRv4_mobile_det --image /paddle/dataset/paddlex/ocr_det/ocr_det_dataset/xxx
+```
+参数解释：
+- `task`: 任务类型，当前支持 `ocrdet`
+- `model`: 模型名称，当前支持 `PP-OCRv4_mobile_det` 和 `PP-OCRv4_mobile_rec`。
+</details>
+
+<details>
+<summary><b> Python API 使用方式</b></summary>
 
 ```python
 import cv2
@@ -23,62 +44,43 @@ draw_img = draw_ocr_box_txt(result['original_image'],result['dt_polys'], result[
 cv2.imwrite("ocr_result.jpg", draw_img[:, :, ::-1], )
 ```
 
-## 图像分类
+参数解释：
+- `task`: 任务类型，当前支持 `ocrdet`
+- `model`: 模型名称，当前支持 `PP-OCRv4_mobile_det` 和 `PP-OCRv4_mobile_rec`。
+</details>
+
+
+## 2.2 图像分类产线
+图像分类产线内置了多个图像分类的单模型，包含 `ResNet` 系列、`PP-LCNet` 系列、`MobileNetV2` 系列、`MobileNetV3` 系列、`ConvNeXt` 系列、`SwinTransformer` 系列、`PP-HGNet` 系列、`PP-HGNetV2` 系列、`CLIP` 系列等模型。具体支持的分类模型列表，您可以参考[模型库](./models/support_model_list.md)，您可以使用以下两种方式进行推理预测，如果在您的场景中，上述模型不能满足您的需求，您可以参考 [PaddleX 模型训练文档](./train/README.md) 进行训练，训练后的模型可以非常方便地集成到该产线中。
+
+<details>
+<summary><b> 命令行使用方式 </b></summary>
+您可以使用命令行将图片的文字识别出来，命令行使用方式如下：
+
+```
+paddlex --task ocrdet --model PP-OCRv4_mobile_det --image /paddle/dataset/paddlex/ocr_det/ocr_det_dataset/xxx
+```
+参数解释：
+- `task`: 任务类型，当前支持 `ocrdet`
+- `model`: 模型名称，当前支持 `PP-OCRv4_mobile_det` 和 `PP-OCRv4_mobile_rec`。
+</details>
+
+
+<details>
+<summary><b> Python API 使用方式</b></summary>
 
 ```python
 from paddlex import ClsPipeline
 from paddlex import PaddleInferenceOption
 
-models = [
-    "ResNet18",
-    "ResNet34",
-    "ResNet50",
-    "ResNet101",
-    "ResNet152",
-    "PP-LCNet_x0_25",
-    "PP-LCNet_x0_35",
-    "PP-LCNet_x0_5",
-    "PP-LCNet_x0_75",
-    "PP-LCNet_x1_0",
-    "PP-LCNet_x1_5",
-    "PP-LCNet_x2_5",
-    "PP-LCNet_x2_0",
-    "MobileNetV3_large_x0_35",
-    "MobileNetV3_large_x0_5",
-    "MobileNetV3_large_x0_75",
-    "MobileNetV3_large_x1_0",
-    "MobileNetV3_large_x1_25",
-    "MobileNetV3_small_x0_35",
-    "MobileNetV3_small_x0_5",
-    "MobileNetV3_small_x0_75",
-    "MobileNetV3_small_x1_0",
-    "MobileNetV3_small_x1_25",
-    "ConvNeXt_tiny",
-    "MobileNetV2_x0_25",
-    "MobileNetV2_x0_5",
-    "MobileNetV2_x1_0",
-    "MobileNetV2_x1_5",
-    "MobileNetV2_x2_0",
-    "SwinTransformer_base_patch4_window7_224",
-    "PP-HGNet_small",
-    "PP-HGNetV2-B0",
-    "PP-HGNetV2-B4",
-    "PP-HGNetV2-B6",
-    "CLIP_vit_base_patch16_224",
-    "CLIP_vit_large_patch14_224",
-]
+pipeline = ClsPipeline(model_name, kernel_option=PaddleInferenceOption())
+    result = pipeline(
+        "/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00006.jpg"
+    )
+    print(result["cls_result"])
+
+</details>
 
-for model_name in models:
-    try:
-        pipeline = ClsPipeline(model_name, kernel_option=PaddleInferenceOption())
-        result = pipeline(
-            "/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00006.jpg"
-        )
-        print(result["cls_result"])
-    except Exception as e:
-        print(f"[ERROR] model: {model_name}; err: {e}")
-    print(f"[INFO] model: {model_name} done!")
-```
 
 ## 目标检测
 

docs/tutorials/wheel.md

@@ -14,7 +14,7 @@ pip install paddlex
 
 ```bash
 cd PaddleX
-python setup.py install
+pip install .
 ```
 
 ### 1.2 安装 PaddleX 相关依赖
@@ -27,21 +27,37 @@ paddlex --install
 
 ### 2.1 使用 CLI 进行推理预测
 
-以图像分类模型 `PP-LCNet_x1_0` 为例，使用inference模型文件（`output/best_model`）对图像（`/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00002.jpg`）进行预测，命令如下：
+以图像分类模型 `PP-LCNet_x1_0` 为例，使用 PaddleX 预置的官方模型对图像（`/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00002.jpg`）进行预测，命令如下：
 
 ```bash
-paddlex --model_name PP-LCNet_x1_0 --model output/best_model --device gpu:0 --input_path /paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00002.jpg --output
+paddlex --pipeline image_classification --model PP-LCNet_x1_0 --input /paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00006.jpg
 ```
 
 可以得到预测结果：
 
 ```
-image_00002.jpg:        class id(s): [76, 24, 70, 18, 14], score(s): [0.66, 0.05, 0.02, 0.02, 0.01], label_name(s): ['tarantula', 'great grey owl, great gray owl, Strix nebulosa', 'harvestman, daddy longlegs, Phalangium opilio', 'magpie', 'indigo bunting, indigo finch, indigo bird, Passerina cyanea']
+[{'class_ids': [309], 'scores': [0.19514], 'label_names': ['bee']}]
 ```
 
+以 OCR 为例，使用PaddleX 预置的 `PP-OCRv4_mobile_det` 和 `PP-OCRv4_mobile_rec` 官方模型，对图像（`/paddle/dataset/paddlex/ocr_det/ocr_det_dataset_examples/images/train_img_100.jpg`）进行预测，命令如下：
+
+```bash
+paddlex --pipeline ocr --model PP-OCRv4_mobile_det PP-OCRv4_mobile_rec --input /paddle/dataset/paddlex/ocr_det/ocr_det_dataset_examples/images/train_img_100.jpg  --output ./
+```
+
+可以在当前目录下得到预测结果示例图 `ocr_result.jpg`。
+
+
 ### 2.2 使用 Python 进行推理预测
 
 ```python
 import paddlex
-paddlex.predict("PP-LCNet_x1_0", "output/best_model", "/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00002.jpg", "gpu", "./output")
+
+model_name = "PP-LCNet_x1_0"
+
+kernel_option = paddlex.PaddleInferenceOption()
+kernel_option.set_device("gpu")
+
+model = paddlex.create_model(model_name, kernel_option=kernel_option)
+model.predict("/paddle/dataset/paddlex/cls/cls_flowers_examples/images/image_00002.jpg")
 ```

paddlex/__init__.py

@@ -17,10 +17,9 @@
 import os
 
 from . import version
-from .paddlex import predict
 from .modules import build_dataset_checker, build_trainer, build_evaluater, build_predictor
+from .modules import create_model, PaddleInferenceOption
 from .pipelines import *
-from .modules.base.predictor.utils.paddle_inference_predictor import PaddleInferenceOption
 
 
 def _initialize():

paddlex/modules/__init__.py

@@ -13,7 +13,8 @@
 # limitations under the License.
 
 
-from .base import build_dataset_checker, build_trainer, build_evaluater, build_predictor, create_model
+from .base import build_dataset_checker, build_trainer, build_evaluater, build_predictor, create_model, \
+PaddleInferenceOption
 from .image_classification import ClsDatasetChecker, ClsTrainer, ClsEvaluator, ClsPredictor
 from .object_detection import COCODatasetChecker, DetTrainer, DetEvaluator, DetPredictor
 from .text_detection import TextDetDatasetChecker, TextDetTrainer, TextDetEvaluator, TextDetPredictor

paddlex/modules/base/__init__.py

@@ -15,7 +15,6 @@
 
 
 from .dataset_checker import build_dataset_checker, BaseDatasetChecker
-from .trainer import build_trainer, BaseTrainer
+from .trainer import build_trainer, BaseTrainer, BaseTrainDeamon
 from .evaluator import build_evaluater, BaseEvaluator
-from .predictor import BasePredictor, BaseTransform, PaddleInferenceOption, build_predictor, create_model
-from .train_deamon import BaseTrainDeamon
+from .predictor import build_predictor, BasePredictor, BaseTransform, PaddleInferenceOption, create_model

paddlex/modules/base/dataset_checker/__init__.py

@@ -46,6 +46,7 @@ class BaseDatasetChecker(ABC, metaclass=AutoRegisterABCMetaClass):
         Args:
             config (AttrDict): PaddleX pipeline config, which is loaded from pipeline yaml file.
         """
+        super().__init__()
         self.global_config = config.Global
         self.check_dataset_config = config.CheckDataset
         self.output_dir = os.path.join(self.global_config.output,

paddlex/modules/base/evaluator.py

@@ -48,6 +48,7 @@ class BaseEvaluator(ABC, metaclass=AutoRegisterABCMetaClass):
         Args:
             config (AttrDict):  PaddleX pipeline config, which is loaded from pipeline yaml file.
         """
+        super().__init__()
         self.global_config = config.Global
         self.eval_config = config.Evaluate
 

paddlex/modules/base/predictor/__init__.py

@@ -16,4 +16,4 @@
 
 from .predictor import BasePredictor, build_predictor, create_model
 from .transform import BaseTransform
-from .utils.paddle_inference_predictor import PaddleInferenceOption
+from .kernel_option import PaddleInferenceOption

paddlex/modules/base/predictor/io/readers.py

@@ -16,9 +16,12 @@
 
 import enum
 import itertools
-
+from pathlib import Path
 import cv2
 
+from .....utils.download import download
+from .....utils.cache import CACHE_DIR
+
 __all__ = ['ImageReader', 'VideoReader', 'ReaderType']
 
 
@@ -62,6 +65,14 @@ class _BaseReader(object):
         """ get default backend arguments """
         return {}
 
+    def _download_from_url(self, in_path):
+        if in_path.startswith("http"):
+            file_name = Path(in_path).name
+            save_path = Path(CACHE_DIR) / "predict_input" / file_name
+            download(in_path, save_path, overwrite=True)
+            return save_path.as_posix()
+        return in_path
+
 
 class ImageReader(_BaseReader):
     """ ImageReader """
@@ -71,6 +82,8 @@ class ImageReader(_BaseReader):
 
     def read(self, in_path):
         """ read the image file from path """
+        # XXX: auto download for url
+        in_path = self._download_from_url(in_path)
         arr = self._backend.read_file(in_path)
         return arr
 

paddlex/modules/base/predictor/kernel_option.py

@@ -0,0 +1,155 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+
+from functools import wraps, partial
+
+
+def register(register_map, key):
+    """register the option setting func
+    """
+
+    def decorator(func):
+        register_map[key] = func
+
+        @wraps(func)
+        def wrapper(self, *args, **kwargs):
+            return func(self, *args, **kwargs)
+
+        return wrapper
+
+    return decorator
+
+
+class PaddleInferenceOption(object):
+    """Paddle Inference Engine Option
+    """
+    SUPPORT_RUN_MODE = ('paddle', 'trt_fp32', 'trt_fp16', 'trt_int8', 'mkldnn',
+                        'mkldnn_bf16')
+    SUPPORT_DEVICE = ('gpu', 'cpu', 'npu', 'xpu', 'mlu')
+    _REGISTER_MAP = {}
+
+    register2self = partial(register, _REGISTER_MAP)
+
+    def __init__(self, **kwargs):
+        super().__init__()
+        self._cfg = {}
+        self._init_option(**kwargs)
+
+    def _init_option(self, **kwargs):
+        for k, v in kwargs.items():
+            if k not in self._REGISTER_MAP:
+                raise Exception(
+                    f"{k} is not supported to set! The supported option is: \
+{list(self._REGISTER_MAP.keys())}")
+            self._REGISTER_MAP.get(k)(self, v)
+        for k, v in self._get_default_config().items():
+            self._cfg.setdefault(k, v)
+
+    def _get_default_config(cls):
+        """ get default config """
+        return {
+            'run_mode': 'paddle',
+            'batch_size': 1,
+            'device': 'gpu',
+            'min_subgraph_size': 3,
+            'shape_info_filename': None,
+            'trt_calib_mode': False,
+            'cpu_threads': 1,
+            'trt_use_static': False
+        }
+
+    @register2self('run_mode')
+    def set_run_mode(self, run_mode: str):
+        """set run mode
+        """
+        if run_mode not in self.SUPPORT_RUN_MODE:
+            support_run_mode_str = ", ".join(self.SUPPORT_RUN_MODE)
+            raise ValueError(
+                f"`run_mode` must be {support_run_mode_str}, but received {repr(run_mode)}."
+            )
+        self._cfg['run_mode'] = run_mode
+
+    @register2self('batch_size')
+    def set_batch_size(self, batch_size: int):
+        """set batch size
+        """
+        if not isinstance(batch_size, int) or batch_size < 1:
+            raise Exception()
+        self._cfg['batch_size'] = batch_size
+
+    @register2self('device')
+    def set_device(self, device: str):
+        """set device
+        """
+        device = device.split(":")[0]
+        if device.lower() not in self.SUPPORT_DEVICE:
+            support_run_mode_str = ", ".join(self.SUPPORT_DEVICE)
+            raise ValueError(
+                f"`device` must be {support_run_mode_str}, but received {repr(device)}."
+            )
+        self._cfg['device'] = device.lower()
+
+    @register2self('min_subgraph_size')
+    def set_min_subgraph_size(self, min_subgraph_size: int):
+        """set min subgraph size
+        """
+        if not isinstance(min_subgraph_size, int):
+            raise Exception()
+        self._cfg['min_subgraph_size'] = min_subgraph_size
+
+    @register2self('shape_info_filename')
+    def set_shape_info_filename(self, shape_info_filename: str):
+        """set shape info filename
+        """
+        self._cfg['shape_info_filename'] = shape_info_filename
+
+    @register2self('trt_calib_mode')
+    def set_trt_calib_mode(self, trt_calib_mode):
+        """set trt calib mode
+        """
+        self._cfg['trt_calib_mode'] = trt_calib_mode
+
+    @register2self('cpu_threads')
+    def set_cpu_threads(self, cpu_threads):
+        """set cpu threads
+        """
+        if not isinstance(cpu_threads, int) or cpu_threads < 1:
+            raise Exception()
+        self._cfg['cpu_threads'] = cpu_threads
+
+    @register2self('trt_use_static')
+    def set_trt_use_static(self, trt_use_static):
+        """set trt use static
+        """
+        self._cfg['trt_use_static'] = trt_use_static
+
+    def get_support_run_mode(self):
+        """get supported run mode
+        """
+        return self.SUPPORT_RUN_MODE
+
+    def get_support_device(self):
+        """get supported device
+        """
+        return self.SUPPORT_DEVICE
+
+    def __str__(self):
+        return ",  ".join([f"{k}: {v}" for k, v in self._cfg.items()])
+
+    def __getattr__(self, key):
+        if key not in self._cfg:
+            raise Exception()
+        return self._cfg.get(key)

paddlex/modules/base/predictor/predictor.py

@@ -18,7 +18,8 @@ import os
 from copy import deepcopy
 from abc import ABC, abstractmethod
 
-from .utils.paddle_inference_predictor import _PaddleInferencePredictor, PaddleInferenceOption
+from .kernel_option import PaddleInferenceOption
+from .utils.paddle_inference_predictor import _PaddleInferencePredictor
 from .utils.mixin import FromDictMixin
 from .utils.batch import batchable_method, Batcher
 from .utils.node import Node
@@ -62,24 +63,20 @@ class BasePredictor(ABC, FromDictMixin, Node):
         if isinstance(input, dict):
             input = [input]
 
-        logging.info(
-            f"Running {self.__class__.__name__}\nModel: {self.model_dir}\nEnv: {self.kernel_option}\n"
-        )
+        logging.debug(
+            f"-------------------- {self.__class__.__name__} --------------------\n\
+Model: {self.model_dir}\nEnv: {self.kernel_option}")
         data = input[0]
         if self.pre_transforms is not None:
             pre_tfs = self.pre_transforms
         else:
             pre_tfs = self._get_pre_transforms_for_data(data)
-        logging.info(
-            f"The following transformation operators will be used for data preprocessing:\n\
-{self._format_transforms(pre_tfs)}\n")
+        logging.debug(f"Preprocess Ops: {self._format_transforms(pre_tfs)}")
         if self.post_transforms is not None:
             post_tfs = self.post_transforms
         else:
             post_tfs = self._get_post_transforms_for_data(data)
-        logging.info(
-            f"The following transformation operators will be used for postprocessing:\n\
-{self._format_transforms(post_tfs)}\n")
+        logging.debug(f"Postprocessing: {self._format_transforms(post_tfs)}")
 
         output = []
         for mini_batch in Batcher(input, batch_size=batch_size):
@@ -132,13 +129,8 @@ class BasePredictor(ABC, FromDictMixin, Node):
 
     def _format_transforms(self, transforms):
         """ format transforms """
-        lines = ['[']
-        for tf in transforms:
-            s = '\t'
-            s += str(tf)
-            lines.append(s)
-        lines.append(']')
-        return '\n'.join(lines)
+        ops_str = ",  ".join([str(tf) for tf in transforms])
+        return f"[{ops_str}]"
 
     def load_other_src(self):
         """ load other source

paddlex/modules/base/predictor/transform.py

@@ -35,3 +35,6 @@ class BaseTransform(FromDictMixin, Node):
     def apply(self, data):
         """ apply """
         raise NotImplementedError
+
+    def __str__(self):
+        return self.__class__.__name__

paddlex/modules/base/predictor/utils/official_models.py

@@ -171,7 +171,7 @@ class OfficialModelsDict(dict):
     def __getitem__(self, key):
         url = super().__getitem__(key)
         save_dir = Path(CACHE_DIR) / "official_models"
-        download_and_extract(url, save_dir, f"{key}")
+        download_and_extract(url, save_dir, f"{key}", overwrite=True)
         return save_dir / f"{key}"
 
 

paddlex/modules/base/predictor/utils/paddle_inference_predictor.py

@@ -14,150 +14,11 @@
 
 
 import os
-from functools import wraps, partial
 from paddle.inference import Config, create_predictor
 
 from .....utils import logging
 
 
-def register(register_map, key):
-    """register the option setting func
-    """
-
-    def decorator(func):
-        register_map[key] = func
-
-        @wraps(func)
-        def wrapper(self, *args, **kwargs):
-            return func(self, *args, **kwargs)
-
-        return wrapper
-
-    return decorator
-
-
-class PaddleInferenceOption(object):
-    """Paddle Inference Engine Option
-    """
-    SUPPORT_RUN_MODE = ('paddle', 'trt_fp32', 'trt_fp16', 'trt_int8', 'mkldnn',
-                        'mkldnn_bf16')
-    SUPPORT_DEVICE = ('gpu', 'cpu', 'npu', 'xpu', 'mlu')
-    _REGISTER_MAP = {}
-
-    register2self = partial(register, _REGISTER_MAP)
-
-    def __init__(self, **kwargs):
-        super().__init__()
-        self._cfg = {}
-        self._init_option(**kwargs)
-
-    def _init_option(self, **kwargs):
-        for k, v in kwargs.items():
-            if k not in self._REGISTER_MAP:
-                raise Exception(
-                    f"{k} is not supported to set! The supported option is: \
-{list(self._REGISTER_MAP.keys())}")
-            self._REGISTER_MAP.get(k)(self, v)
-        for k, v in self._get_default_config().items():
-            self._cfg.setdefault(k, v)
-
-    def _get_default_config(cls):
-        """ get default config """
-        return {
-            'run_mode': 'paddle',
-            'batch_size': 1,
-            'device': 'gpu',
-            'min_subgraph_size': 3,
-            'shape_info_filename': None,
-            'trt_calib_mode': False,
-            'cpu_threads': 1,
-            'trt_use_static': False
-        }
-
-    @register2self('run_mode')
-    def set_run_mode(self, run_mode: str):
-        """set run mode
-        """
-        if run_mode not in self.SUPPORT_RUN_MODE:
-            support_run_mode_str = ", ".join(self.SUPPORT_RUN_MODE)
-            raise ValueError(
-                f"`run_mode` must be {support_run_mode_str}, but received {repr(run_mode)}."
-            )
-        self._cfg['run_mode'] = run_mode
-
-    @register2self('batch_size')
-    def set_batch_size(self, batch_size: int):
-        """set batch size
-        """
-        if not isinstance(batch_size, int) or batch_size < 1:
-            raise Exception()
-        self._cfg['batch_size'] = batch_size
-
-    @register2self('device')
-    def set_device(self, device: str):
-        """set device
-        """
-        device = device.split(":")[0]
-        if device.lower() not in self.SUPPORT_DEVICE:
-            support_run_mode_str = ", ".join(self.SUPPORT_DEVICE)
-            raise ValueError(
-                f"`device` must be {support_run_mode_str}, but received {repr(device)}."
-            )
-        self._cfg['device'] = device.lower()
-
-    @register2self('min_subgraph_size')
-    def set_min_subgraph_size(self, min_subgraph_size: int):
-        """set min subgraph size
-        """
-        if not isinstance(min_subgraph_size, int):
-            raise Exception()
-        self._cfg['min_subgraph_size'] = min_subgraph_size
-
-    @register2self('shape_info_filename')
-    def set_shape_info_filename(self, shape_info_filename: str):
-        """set shape info filename
-        """
-        self._cfg['shape_info_filename'] = shape_info_filename
-
-    @register2self('trt_calib_mode')
-    def set_trt_calib_mode(self, trt_calib_mode):
-        """set trt calib mode
-        """
-        self._cfg['trt_calib_mode'] = trt_calib_mode
-
-    @register2self('cpu_threads')
-    def set_cpu_threads(self, cpu_threads):
-        """set cpu threads
-        """
-        if not isinstance(cpu_threads, int) or cpu_threads < 1:
-            raise Exception()
-        self._cfg['cpu_threads'] = cpu_threads
-
-    @register2self('trt_use_static')
-    def set_trt_use_static(self, trt_use_static):
-        """set trt use static
-        """
-        self._cfg['trt_use_static'] = trt_use_static
-
-    def get_support_run_mode(self):
-        """get supported run mode
-        """
-        return self.SUPPORT_RUN_MODE
-
-    def get_support_device(self):
-        """get supported device
-        """
-        return self.SUPPORT_DEVICE
-
-    def __str__(self):
-        return "\n  " + "\n  ".join([f"{k}: {v}" for k, v in self._cfg.items()])
-
-    def __getattr__(self, key):
-        if key not in self._cfg:
-            raise Exception()
-        return self._cfg.get(key)
-
-
 class _PaddleInferencePredictor(object):
     """ Predictor based on Paddle Inference """
 
@@ -183,10 +44,14 @@ self._create(param_path, model_path, option, delete_pass=delete_pass)
             config.enable_use_gpu(200, 0)
         elif option.device == 'npu':
             config.enable_custom_device('npu')
+            os.environ["FLAGS_npu_jit_compile"] = 0
+            os.environ["FLAGS_use_stride_kernel"] = 0
+            os.environ["FLAGS_allocator_strategy"] = "auto_growth"
         elif option.device == 'xpu':
             config.enable_custom_device('npu')
         elif option.device == 'mlu':
             config.enable_custom_device('mlu')
+            os.environ["CUSTOM_DEVICE_BLACK_LIST"] = "set_value"
         else:
             assert option.device == 'cpu'
             config.disable_gpu()

paddlex/modules/base/trainer/__init__.py

@@ -0,0 +1,18 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+
+from .trainer import build_trainer, BaseTrainer
+from .train_deamon import BaseTrainDeamon

paddlex/modules/base/train_deamon.py → paddlex/modules/base/trainer/train_deamon.py

@@ -21,8 +21,9 @@ import traceback
 import threading
 from abc import ABC, abstractmethod
 from pathlib import Path
-from .build_model import build_model
-from ...utils.file_interface import write_json_file
+from ..build_model import build_model
+from ....utils.file_interface import write_json_file
+from ....utils import logging
 
 
 def try_except_decorator(func):
@@ -34,7 +35,7 @@ def try_except_decorator(func):
         except Exception as e:
             exc_type, exc_value, exc_tb = sys.exc_info()
             self.save_json()
-            traceback.print_exception(exc_type, exc_value, exc_tb)
+            traceback.logging.info_exception(exc_type, exc_value, exc_tb)
         finally:
             self.processing = False
 

paddlex/modules/base/trainer.py → paddlex/modules/base/trainer/trainer.py

@@ -16,10 +16,10 @@
 import os
 from abc import ABC, abstractmethod
 from pathlib import Path
-from .build_model import build_model
-from ...utils.device import get_device
-from ...utils.misc import AutoRegisterABCMetaClass
-from ...utils.config import AttrDict
+from ..build_model import build_model
+from ....utils.device import get_device
+from ....utils.misc import AutoRegisterABCMetaClass
+from ....utils.config import AttrDict
 
 
 def build_trainer(config: AttrDict) -> "BaseTrainer":
@@ -45,6 +45,7 @@ class BaseTrainer(ABC, metaclass=AutoRegisterABCMetaClass):
         Args:
             config (AttrDict):  PaddleX pipeline config, which is loaded from pipeline yaml file.
         """
+        super().__init__()
         self.global_config = config.Global
         self.train_config = config.Train
 

paddlex/modules/base/utils/topk_eval.py

@@ -19,6 +19,8 @@ import argparse
 from paddle import nn
 import paddle
 
+from ....utils import logging
+
 
 def parse_args():
     """Parse all arguments """
@@ -77,7 +79,7 @@ class TopkAcc(AvgMetrics):
             if output_dims < k:
                 if not self.warned:
                     msg = f"The output dims({output_dims}) is less than k({k}), so the Top-{k} metric is meaningless."
-                    print(msg)
+                    logging.info(msg)
                     self.warned = True
                 metric_dict[f"top{k}"] = 1
             else:
@@ -111,7 +113,7 @@ def main(args):
         pred.append(prase_pt_info(pt_info, args.num_classes))
         label.append([gt_info[img_file]])
     metric_dict = TopkAcc()(paddle.to_tensor(pred), paddle.to_tensor(label))
-    print(metric_dict)
+    logging.info(metric_dict)
 
 
 if __name__ == "__main__":

paddlex/modules/image_classification/__init__.py

@@ -16,4 +16,4 @@
 from .trainer import ClsTrainer
 from .dataset_checker import ClsDatasetChecker
 from .evaluator import ClsEvaluator
-from .predictor import ClsPredictor
+from .predictor import ClsPredictor, transforms

paddlex/modules/image_classification/dataset_checker/__init__.py

@@ -15,7 +15,7 @@
 
 from pathlib import Path
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, split_dataset, deep_analyse
 from ..support_models import SUPPORT_MODELS
 

paddlex/modules/image_classification/evaluator.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/image_classification/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import ClsPredictor
+from . import transforms

paddlex/modules/image_classification/trainer.py

@@ -18,8 +18,7 @@ import shutil
 import paddle
 from pathlib import Path
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from .support_models import SUPPORT_MODELS
 from ...utils.config import AttrDict
 

paddlex/modules/instance_segmentation/__init__.py

@@ -16,4 +16,4 @@
 from .dataset_checker import COCOInstSegDatasetChecker
 from .trainer import InstanceSegTrainer
 from .evaluator import InstanceSegEvaluator
-from .predictor import InstanceSegPredictor
+from .predictor import InstanceSegPredictor, transforms

paddlex/modules/instance_segmentation/dataset_checker/__init__.py

@@ -16,7 +16,7 @@
 import os
 
 from .dataset_src import check, convert, split_dataset, deep_analyse
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 
 from ..support_models import SUPPORT_MODELS
 

paddlex/modules/instance_segmentation/dataset_checker/dataset_src/utils/visualizer.py

@@ -21,6 +21,7 @@ from PIL import Image, ImageDraw, ImageFont
 from pycocotools.coco import COCO
 
 from ......utils.fonts import PINGFANG_FONT_FILE_PATH
+from ......utils import logging
 
 
 def colormap(rgb=False):
@@ -113,7 +114,7 @@ def draw_bbox(image, coco_info: COCO, img_id):
             xmin = min(x1, x2, x3, x4)
             ymin = min(y1, y2, y3, y4)
         else:
-            print('Error: The shape of bbox must be [M, 4] or [M, 8]!')
+            logging.info('Error: The shape of bbox must be [M, 4] or [M, 8]!')
 
         # draw label
         label = coco_info.loadCats(catid)[0]['name']

paddlex/modules/instance_segmentation/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import InstanceSegPredictor
+from . import transforms

paddlex/modules/object_detection/__init__.py

@@ -16,4 +16,4 @@
 from .trainer import DetTrainer
 from .dataset_checker import COCODatasetChecker
 from .evaluator import DetEvaluator
-from .predictor import DetPredictor
+from .predictor import DetPredictor, transforms

paddlex/modules/object_detection/dataset_checker/__init__.py

@@ -22,7 +22,7 @@ from PIL import Image
 import json
 from pycocotools.coco import COCO
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, convert, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/object_detection/dataset_checker/dataset_src/utils/visualizer.py

@@ -15,6 +15,7 @@ from PIL import Image, ImageDraw, ImageFont
 from pycocotools.coco import COCO
 
 from ......utils.fonts import PINGFANG_FONT_FILE_PATH
+from ......utils import logging
 
 
 def colormap(rgb=False):
@@ -107,7 +108,7 @@ def draw_bbox(image, coco_info: COCO, img_id):
             xmin = min(x1, x2, x3, x4)
             ymin = min(y1, y2, y3, y4)
         else:
-            print('Error: The shape of bbox must be [M, 4] or [M, 8]!')
+            logging.info('Error: The shape of bbox must be [M, 4] or [M, 8]!')
 
         # draw label
         label = coco_info.loadCats(catid)[0]['name']

paddlex/modules/object_detection/evaluator.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/object_detection/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import DetPredictor
+from . import transforms

paddlex/modules/object_detection/trainer.py

@@ -16,9 +16,9 @@
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
+from ...utils import logging
 from .support_models import SUPPORT_MODELS
 
 
@@ -77,7 +77,7 @@ class DetTrainer(BaseTrainer):
             try:
                 self.pdx_config.update_static_assigner_epochs(assigner_epochs)
             except Exception:
-                print(
+                logging.info(
                     f"The model({self.global_config.model}) don't support to update_static_assigner_epochs!"
                 )
 

paddlex/modules/semantic_segmentation/__init__.py

@@ -16,4 +16,4 @@
 from .dataset_checker import SegDatasetChecker
 from .trainer import SegTrainer
 from .evaluator import SegEvaluator
-from .predictor import SegPredictor
+from .predictor import SegPredictor, transforms

paddlex/modules/semantic_segmentation/dataset_checker/__init__.py

@@ -16,7 +16,7 @@
 import os
 import os.path as osp
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check_dataset, convert_dataset, split_dataset, anaylse_dataset
 
 from ..support_models import SUPPORT_MODELS
@@ -36,10 +36,8 @@ class SegDatasetChecker(BaseDatasetChecker):
         Returns:
             str: the root directory of converted dataset.
         """
-        converted_dataset_path = osp.join(
-            os.getenv("TEMP_DIR"), "converted_dataset")
         return convert_dataset(self.check_dataset_config.src_dataset_type,
-                               dataset_dir, converted_dataset_path)
+                               src_dataset_dir)
 
     def split_dataset(self, src_dataset_dir: str) -> str:
         """repartition the train and validation dataset
@@ -50,9 +48,9 @@ class SegDatasetChecker(BaseDatasetChecker):
         Returns:
             str: the root directory of splited dataset.
         """
-        return split_dataset(dataset_dir,
-                             self.check_dataset_config.split_train_percent,
-                             self.check_dataset_config.split_val_percent)
+        return split_dataset(src_dataset_dir,
+                             self.check_dataset_config.split.train_percent,
+                             self.check_dataset_config.split.val_percent)
 
     def check_dataset(self, dataset_dir: str,
                       sample_num: int=sample_num) -> dict:
@@ -64,7 +62,7 @@ class SegDatasetChecker(BaseDatasetChecker):
         Returns:
             dict: dataset summary.
         """
-        return check_dataset(dataset_dir, self.global_config.output, sample_num)
+        return check_dataset(dataset_dir, self.output_dir, sample_num)
 
     def analyse(self, dataset_dir: str) -> dict:
         """deep analyse dataset
@@ -75,7 +73,7 @@ class SegDatasetChecker(BaseDatasetChecker):
         Returns:
             dict: the deep analysis results.
         """
-        return anaylse_dataset(dataset_dir, self.global_config.output)
+        return anaylse_dataset(dataset_dir, self.output_dir)
 
     def get_show_type(self) -> str:
         """get the show type of dataset

paddlex/modules/semantic_segmentation/dataset_checker/dataset_src/analyse_dataset.py

@@ -22,20 +22,16 @@ import numpy as np
 from PIL import Image, ImageOps
 
 from .....utils.file_interface import custom_open
+from .....utils.logging import info
 
 
 def anaylse_dataset(dataset_dir, output_dir):
     """class analysis for dataset"""
 
-    split_tags = ["train", "val", "test"]
+    split_tags = ["train", "val"]
     label2count = {tag: dict() for tag in split_tags}
     for tag in split_tags:
         mapping_file = osp.join(dataset_dir, f"{tag}.txt")
-        if not osp.exists(mapping_file) and (tag == "test"):
-            print(
-                f"The mapping file ({mapping_file}) doesn't exist, will be ignored."
-            )
-            continue
         with custom_open(mapping_file, "r") as fp:
             lines = filter(None, (line.strip() for line in fp.readlines()))
             for i, line in enumerate(lines):
@@ -66,22 +62,15 @@ def anaylse_dataset(dataset_dir, output_dir):
     fig, ax = plt.subplots(
         figsize=(max(8, int(len(label_idx) / 5)), 5), dpi=120)
 
-    width = 0.333 if label2count["test"] else 0.5,
+    width = 0.5,
     ax.bar(x, train_list, width=width, label="train")
     ax.bar(x + width, val_list, width=width, label="val")
 
-    if label2count["test"]:
-        ax.bar(x + 2 * width,
-               list(label2count["test"].values()),
-               width=0.33,
-               label="test")
-        plt.xticks(x + 0.33, label_idx)
-    else:
-        plt.xticks(x + 0.25, label_idx)
+    plt.xticks(x + 0.25, label_idx)
     ax.set_xlabel('Label Index')
     ax.set_ylabel('Sample Counts')
     plt.legend()
     fig.tight_layout()
     fig_path = os.path.join(output_dir, "histogram.png")
     fig.savefig(fig_path)
-    return {"histogram": "histogram.png"}
+    return {"histogram": os.path.join("check_dataset", "histogram.png")}

paddlex/modules/semantic_segmentation/dataset_checker/dataset_src/check_dataset.py

@@ -23,6 +23,7 @@ from PIL import Image, ImageOps
 from .utils.visualizer import visualize
 from .....utils.errors import DatasetFileNotFoundError
 from .....utils.file_interface import custom_open
+from .....utils.logging import info
 
 
 def check_dataset(dataset_dir, output_dir, sample_num=10):
@@ -30,7 +31,7 @@ def check_dataset(dataset_dir, output_dir, sample_num=10):
     dataset_dir = osp.abspath(dataset_dir)
     if not osp.exists(dataset_dir) or not osp.isdir(dataset_dir):
         raise DatasetFileNotFoundError(file_path=dataset_dir)
-    vis_save_dir = osp.join(output_dir, 'tmp')
+    vis_save_dir = osp.join(output_dir, 'demo_img')
     if not osp.exists(vis_save_dir):
         os.makedirs(vis_save_dir)
     split_tags = ["train", "val"]
@@ -39,7 +40,7 @@ def check_dataset(dataset_dir, output_dir, sample_num=10):
     for tag in split_tags:
         mapping_file = osp.join(dataset_dir, f"{tag}.txt")
         if not osp.exists(mapping_file):
-            print(f"The mapping file ({mapping_file}) doesn't exist, ignored.")
+            info(f"The mapping file ({mapping_file}) doesn't exist, ignored.")
             continue
         with custom_open(mapping_file, "r") as fp:
             lines = filter(None, (line.strip() for line in fp.readlines()))
@@ -65,6 +66,9 @@ def check_dataset(dataset_dir, output_dir, sample_num=10):
                     vis_save_path = osp.join(vis_save_dir,
                                              osp.basename(img_file))
                     vis_img.save(vis_save_path)
+                    vis_save_path = osp.join(
+                        'check_dataset',
+                        os.path.relpath(vis_save_path, output_dir))
                     if f"{tag}_sample_paths" not in attrs:
                         attrs[f"{tag}_sample_paths"] = [vis_save_path]
                     else:

paddlex/modules/semantic_segmentation/dataset_checker/dataset_src/convert_dataset.py

@@ -24,41 +24,36 @@ import numpy as np
 from PIL import Image, ImageDraw
 
 from .....utils.file_interface import custom_open
+from .....utils import logging
 
 
-def convert_dataset(dataset_type, input_dir, output_dir):
+def convert_dataset(dataset_type, input_dir):
     """ convert to paddlex official format """
     if dataset_type == "LabelMe":
-        return convert_labelme_dataset(input_dir, output_dir)
+        return convert_labelme_dataset(input_dir)
     else:
         raise NotImplementedError(dataset_type)
 
 
-def convert_labelme_dataset(input_dir, output_dir):
+def convert_labelme_dataset(input_dir):
     """ convert labelme format to paddlex official format"""
     bg_name = "_background_"
     ignore_name = '__ignore__'
 
     # prepare dir
-    output_img_dir = osp.join(output_dir, 'images')
-    output_annot_dir = osp.join(output_dir, 'annotations')
-    if not osp.exists(output_dir):
-        os.makedirs(output_dir)
-    if osp.exists(output_img_dir):
-        shutil.rmtree(output_img_dir)
-    os.makedirs(output_img_dir)
-    if osp.exists(output_annot_dir):
-        shutil.rmtree(output_annot_dir)
-    os.makedirs(output_annot_dir)
+    output_img_dir = osp.join(input_dir, 'images')
+    output_annot_dir = osp.join(input_dir, 'annotations')
+    if not osp.exists(output_img_dir):
+        os.makedirs(output_img_dir)
+    if not osp.exists(output_annot_dir):
+        os.makedirs(output_annot_dir)
 
     # collect class_names and set class_name_to_id
     class_names = []
     class_name_to_id = {}
-    split_tags = ["train", "val", "test"]
+    split_tags = ["train", "val"]
     for tag in split_tags:
         mapping_file = osp.join(input_dir, f'{tag}_anno_list.txt')
-        if not osp.exists(mapping_file) and (tag == "test"):
-            continue
         with open(mapping_file, 'r') as f:
             label_files = [
                 osp.join(input_dir, line.strip('\n')) for line in f.readlines()
@@ -96,7 +91,8 @@ def convert_labelme_dataset(input_dir, output_dir):
                 data = json.load(f)
                 img_path = osp.join(osp.dirname(label_file), data['imagePath'])
                 if not os.path.exists(img_path):
-                    print('%s is not existed, skip this image' % img_path)
+                    logging.info('%s is not existed, skip this image' %
+                                 img_path)
                     continue
                 img_name = img_path.split('/')[-1]
                 img_file_list.append(f"images/{img_name}")
@@ -113,18 +109,18 @@ def convert_labelme_dataset(input_dir, output_dir):
                 lbl_pil.save(annotated_img_path)
 
                 shutil.copy(img_path, output_img_dir)
-        with custom_open(osp.join(output_dir, f'{tag}.txt'), 'w') as fp:
+        with custom_open(osp.join(input_dir, f'{tag}.txt'), 'w') as fp:
             for img_path, lbl_path in zip(img_file_list, label_file_list):
                 fp.write(f'{img_path} {lbl_path}\n')
 
-    with custom_open(osp.join(output_dir, 'class_name.txt'), 'w') as fp:
+    with custom_open(osp.join(input_dir, 'class_name.txt'), 'w') as fp:
         for name in class_names:
             fp.write(f'{name}{os.linesep}')
-    with custom_open(osp.join(output_dir, 'class_name_to_id.txt'), 'w') as fp:
+    with custom_open(osp.join(input_dir, 'class_name_to_id.txt'), 'w') as fp:
         for key, val in class_name_to_id.items():
             fp.write(f'{val}: {key}{os.linesep}')
 
-    return output_dir
+    return input_dir
 
 
 def get_color_map_list(num_classes):

paddlex/modules/semantic_segmentation/dataset_checker/dataset_src/split_dataset.py

@@ -20,18 +20,19 @@ import random
 import shutil
 
 from .....utils.file_interface import custom_open
+from .....utils import logging
 
 
-def split_dataset(root_dir, train_percent, val_percent, test_percent=0):
+def split_dataset(root_dir, train_percent, val_percent):
     """ split dataset """
     assert train_percent > 0, ValueError(
         f"The train_percent({train_percent}) must greater than 0!")
     assert val_percent > 0, ValueError(
         f"The val_percent({val_percent}) must greater than 0!")
-    if train_percent + val_percent + test_percent != 100:
+    if train_percent + val_percent != 100:
         raise ValueError(
-            f"The sum of train_percent({train_percent}), val_percent({val_percent}) "
-            f"and test_percent({test_percent}) should be 100!")
+            f"The sum of train_percent({train_percent})and val_percent({val_percent}) should be 100!"
+        )
 
     img_dir = osp.join(root_dir, "images")
     assert osp.exists(img_dir), FileNotFoundError(
@@ -51,12 +52,13 @@ def split_dataset(root_dir, train_percent, val_percent, test_percent=0):
     assert img_num == ann_num, ValueError(
         "The number of images and annotations must be equal!")
 
-    split_tags = ["train", "val", "test"]
+    split_tags = ["train", "val"]
     mapping_line_list = []
     for tag in split_tags:
         mapping_file = osp.join(root_dir, f"{tag}.txt")
         if not osp.exists(mapping_file):
-            print(f"The mapping file ({mapping_file}) doesn't exist, ignored.")
+            logging.info(
+                f"The mapping file ({mapping_file}) doesn't exist, ignored.")
             continue
         with custom_open(mapping_file, "r") as fp:
             lines = filter(None, (line.strip() for line in fp.readlines()))
@@ -64,7 +66,7 @@ def split_dataset(root_dir, train_percent, val_percent, test_percent=0):
 
     sample_num = len(mapping_line_list)
     random.shuffle(mapping_line_list)
-    split_percents = [train_percent, val_percent, test_percent]
+    split_percents = [train_percent, val_percent]
     start_idx = 0
     for tag, percent in zip(split_tags, split_percents):
         if tag == 'test' and percent == 0:
@@ -74,8 +76,8 @@ def split_dataset(root_dir, train_percent, val_percent, test_percent=0):
         mapping_file = osp.join(root_dir, f"{tag}.txt")
         if os.path.exists(mapping_file):
             shutil.move(mapping_file, mapping_file + ".bak")
-            print(f"The original mapping file ({mapping_file}) "
-                  f"has been backed up to ({mapping_file}.bak)")
+            logging.info(f"The original mapping file ({mapping_file}) "
+                         f"has been backed up to ({mapping_file}.bak)")
         with custom_open(mapping_file, "w") as fp:
             fp.write("\n".join(mapping_line_list[start_idx:end_idx]))
         start_idx = end_idx

paddlex/modules/semantic_segmentation/evaluator.py

@@ -14,7 +14,7 @@
 
 
 from pathlib import Path
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/semantic_segmentation/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import SegPredictor
+from . import transforms

paddlex/modules/semantic_segmentation/trainer.py

@@ -18,8 +18,7 @@ import glob
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/table_recognition/__init__.py

@@ -16,4 +16,4 @@
 from .dataset_checker import TableRecDatasetChecker
 from .trainer import TableRecTrainer
 from .evaluator import TableRecEvaluator
-from .predictor import TableRecPredictor
+from .predictor import TableRecPredictor, transforms

paddlex/modules/table_recognition/dataset_checker/__init__.py

@@ -19,7 +19,7 @@ from collections import defaultdict, Counter
 from PIL import Image
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/table_recognition/evaluator.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/table_recognition/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import TableRecPredictor
+from . import transforms

paddlex/modules/table_recognition/trainer.py

@@ -18,8 +18,7 @@ import os
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/text_detection/__init__.py

@@ -16,4 +16,4 @@
 from .dataset_checker import TextDetDatasetChecker
 from .trainer import TextDetTrainer
 from .evaluator import TextDetEvaluator
-from .predictor import TextDetPredictor
+from .predictor import TextDetPredictor, transforms

paddlex/modules/text_detection/dataset_checker/__init__.py

@@ -19,7 +19,7 @@ from collections import defaultdict, Counter
 
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/text_detection/evaluator.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/text_detection/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import TextDetPredictor
+from . import transforms

paddlex/modules/text_detection/predictor/transforms.py

@@ -23,6 +23,7 @@ import numpy as np
 from PIL import Image
 from shapely.geometry import Polygon
 
+from ....utils import logging
 from ...base.predictor.io.writers import ImageWriter
 from ...base.predictor.io.readers import ImageReader
 from ...base.predictor import BaseTransform
@@ -150,7 +151,7 @@ class DetResizeForTest(BaseTransform):
                 return None, (None, None)
             img = cv2.resize(img, (int(resize_w), int(resize_h)))
         except:
-            print(img.shape, resize_w, resize_h)
+            logging.info(img.shape, resize_w, resize_h)
             sys.exit(0)
         ratio_h = resize_h / float(h)
         ratio_w = resize_w / float(w)

paddlex/modules/text_detection/trainer.py

@@ -17,8 +17,7 @@ import os
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/text_recognition/__init__.py

@@ -16,4 +16,4 @@
 from .dataset_checker import TextRecDatasetChecker
 from .trainer import TextRecTrainer
 from .evaluator import TextRecEvaluator
-from .predictor import TextRecPredictor
+from .predictor import TextRecPredictor, transforms

paddlex/modules/text_recognition/dataset_checker/__init__.py

@@ -20,7 +20,7 @@ from collections import defaultdict, Counter
 from PIL import Image
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/text_recognition/evaluator.py

@@ -15,7 +15,7 @@
 
 from pathlib import Path
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/text_recognition/predictor/__init__.py

@@ -15,3 +15,4 @@
 
 
 from .predictor import TextRecPredictor
+from . import transforms

paddlex/modules/text_recognition/predictor/predictor.py

@@ -23,6 +23,7 @@ from ...base.predictor.transforms import image_common
 from ...base import BasePredictor
 from .keys import TextRecKeys as K
 from . import transforms as T
+from .utils import InnerConfig
 from ..support_models import SUPPORT_MODELS
 
 
@@ -30,6 +31,14 @@ class TextRecPredictor(BasePredictor):
     """ TextRecPredictor """
     support_models = SUPPORT_MODELS
 
+    def load_other_src(self):
+        """ load the inner config file """
+        infer_cfg_file_path = os.path.join(self.model_dir, 'inference.yml')
+        if not os.path.exists(infer_cfg_file_path):
+            raise FileNotFoundError(
+                f"Cannot find config file: {infer_cfg_file_path}")
+        return InnerConfig(infer_cfg_file_path)
+
     @classmethod
     def get_input_keys(cls):
         """ get input keys """
@@ -73,8 +82,8 @@ class TextRecPredictor(BasePredictor):
 
     def _get_post_transforms_for_data(self, data):
         """ get postprocess transforms """
-        post_transforms = [T.CTCLabelDecode()]
+        post_transforms = [T.CTCLabelDecode(self.other_src.PostProcess)]
         if data.get('cli_flag', False):
             output_dir = data.get("output_dir", "./")
             post_transforms.append(T.PrintResult())
-        return post_transforms
+        return post_transforms

paddlex/modules/text_recognition/predictor/transforms.py

@@ -83,29 +83,9 @@ class OCRReisizeNormImg(BaseTransform):
 class BaseRecLabelDecode(BaseTransform):
     """ Convert between text-label and text-index """
 
-    def __init__(self, character_dict_type='ch', use_space_char=True):
+    def __init__(self, character_str=None):
         self.reverse = False
-        self.character_str = []
-
-        if character_dict_type is None:
-            self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz"
-            dict_character = list(self.character_str)
-        elif character_dict_type == 'ch':
-            character_dict_name = 'ppocr_keys_v1.txt'
-            character_dict_path = osp.abspath(
-                osp.join(osp.dirname(__file__), character_dict_name))
-            with open(character_dict_path, "rb") as fin:
-                lines = fin.readlines()
-                for line in lines:
-                    line = line.decode('utf-8').strip("\n").strip("\r\n")
-                    self.character_str.append(line)
-            if use_space_char:
-                self.character_str.append(" ")
-            dict_character = list(self.character_str)
-            if 'arabic' in character_dict_path:
-                self.reverse = True
-        else:
-            assert False, " character_dict_type must be 'ch' or None "
+        dict_character = character_str if character_str is not None else "0123456789abcdefghijklmnopqrstuvwxyz"
 
         dict_character = self.add_special_char(dict_character)
         self.dict = {}
@@ -199,8 +179,10 @@ class BaseRecLabelDecode(BaseTransform):
 class CTCLabelDecode(BaseRecLabelDecode):
     """ Convert between text-label and text-index """
 
-    def __init__(self, character_dict_type='ch', use_space_char=True):
-        super().__init__(character_dict_type, use_space_char)
+    def __init__(self, post_process_cfg=None):
+        assert post_process_cfg['name'] == 'CTCLabelDecode'
+        character_str = post_process_cfg['character_dict']
+        super().__init__(character_str)
 
     def apply(self, data):
         """ apply """

paddlex/modules/text_recognition/predictor/utils.py

@@ -22,52 +22,67 @@ from ....utils import logging
 from ...base.predictor.transforms import image_common
 
 
-def read_pre_transforms_from_file(config_path):
-    """ read preprocess transforms from  config file """
+class InnerConfig(object):
+    """Inner Config"""
 
-    def _process_incompct_args(cfg, arg_names, action):
-        """ process incompct arguments """
-        for name in arg_names:
-            if name in cfg:
-                if action == 'ignore':
-                    logging.warning(f"Ignoring incompatible argument: {name}")
-                elif action == 'raise':
-                    raise RuntimeError(
-                        f"Incompatible argument detected: {name}")
-                else:
-                    raise ValueError(f"Unknown action: {action}")
+    def __init__(self, config_path):
+        self.inner_cfg = self.load(config_path)
 
-    with codecs.open(config_path, 'r', 'utf-8') as file:
-        dic = yaml.load(file, Loader=yaml.FullLoader)
-    tfs_cfg = dic['Deploy']['transforms']
-    tfs = []
-    for cfg in tfs_cfg:
-        if cfg['type'] == 'Normalize':
-            tf = image_common.Normalize(
-                mean=cfg.get('mean', 0.5), std=cfg.get('std', 0.5))
-        elif cfg['type'] == 'Resize':
-            tf = image_common.Resize(
-                target_size=cfg.get('target_size', (512, 512)),
-                keep_ratio=cfg.get('keep_ratio', False),
-                size_divisor=cfg.get('size_divisor', None),
-                interp=cfg.get('interp', 'LINEAR'))
-        elif cfg['type'] == 'ResizeByLong':
-            tf = image_common.ResizeByLong(
-                target_long_edge=cfg['long_size'],
-                size_divisor=None,
-                interp='LINEAR')
-        elif cfg['type'] == 'ResizeByShort':
-            _process_incompct_args(cfg, ['max_size'], action='raise')
-            tf = image_common.ResizeByShort(
-                target_short_edge=cfg['short_size'],
-                size_divisor=None,
-                interp='LINEAR')
-        elif cfg['type'] == 'Padding':
-            _process_incompct_args(
-                cfg, ['label_padding_value'], action='ignore')
-            tf = image_common.Pad(target_size=cfg['target_size'],
-                                  val=cfg.get('im_padding_value', 127.5))
-        else:
-            raise RuntimeError(f"Unsupported type: {cfg['type']}")
-        tfs.append(tf)
-    return tfs
+    def load(self, config_path):
+        """ load infer config """
+        with codecs.open(config_path, 'r', 'utf-8') as file:
+            dic = yaml.load(file, Loader=yaml.FullLoader)
+        return dic
+
+    @property
+    def pre_transforms(self):
+        """ read preprocess transforms from  config file """
+
+        def _process_incompct_args(cfg, arg_names, action):
+            """ process incompct arguments """
+            for name in arg_names:
+                if name in cfg:
+                    if action == 'ignore':
+                        logging.warning(
+                            f"Ignoring incompatible argument: {name}")
+                    elif action == 'raise':
+                        raise RuntimeError(
+                            f"Incompatible argument detected: {name}")
+                    else:
+                        raise ValueError(f"Unknown action: {action}")
+
+        tfs_cfg = self.inner_cfg['Deploy']['transforms']
+        tfs = []
+        for cfg in tfs_cfg:
+            if cfg['type'] == 'Normalize':
+                tf = image_common.Normalize(
+                    mean=cfg.get('mean', 0.5), std=cfg.get('std', 0.5))
+            elif cfg['type'] == 'Resize':
+                tf = image_common.Resize(
+                    target_size=cfg.get('target_size', (512, 512)),
+                    keep_ratio=cfg.get('keep_ratio', False),
+                    size_divisor=cfg.get('size_divisor', None),
+                    interp=cfg.get('interp', 'LINEAR'))
+            elif cfg['type'] == 'ResizeByLong':
+                tf = image_common.ResizeByLong(
+                    target_long_edge=cfg['long_size'],
+                    size_divisor=None,
+                    interp='LINEAR')
+            elif cfg['type'] == 'ResizeByShort':
+                _process_incompct_args(cfg, ['max_size'], action='raise')
+                tf = image_common.ResizeByShort(
+                    target_short_edge=cfg['short_size'],
+                    size_divisor=None,
+                    interp='LINEAR')
+            elif cfg['type'] == 'Padding':
+                _process_incompct_args(
+                    cfg, ['label_padding_value'], action='ignore')
+                tf = image_common.Pad(target_size=cfg['target_size'],
+                                      val=cfg.get('im_padding_value', 127.5))
+            else:
+                raise RuntimeError(f"Unsupported type: {cfg['type']}")
+            tfs.append(tf)
+        return tfs
+
+    def __getattr__(self, key):
+        return self.inner_cfg[key]

paddlex/modules/text_recognition/trainer.py

@@ -18,8 +18,7 @@ import shutil
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/ts_anomaly_detection/dataset_checker/__init__.py

@@ -20,7 +20,7 @@ from collections import defaultdict, Counter
 from PIL import Image
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, convert, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/ts_anomaly_detection/evaluator.py

@@ -16,7 +16,7 @@
 import tarfile
 from pathlib import Path
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/ts_anomaly_detection/trainer.py

@@ -20,8 +20,7 @@ from pathlib import Path
 import tarfile
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/ts_classification/dataset_checker/__init__.py

@@ -20,7 +20,7 @@ from collections import defaultdict, Counter
 from PIL import Image
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, convert, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/ts_classification/evaluator.py

@@ -16,7 +16,7 @@
 import tarfile
 from pathlib import Path
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/ts_classification/trainer.py

@@ -20,8 +20,7 @@ import tarfile
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/modules/ts_forecast/dataset_checker/__init__.py

@@ -20,7 +20,7 @@ from collections import defaultdict, Counter
 from PIL import Image
 import json
 
-from ...base.dataset_checker import BaseDatasetChecker
+from ...base import BaseDatasetChecker
 from .dataset_src import check, convert, split_dataset, deep_analyse
 
 from ..support_models import SUPPORT_MODELS

paddlex/modules/ts_forecast/evaluator.py

@@ -16,7 +16,7 @@
 import tarfile
 from pathlib import Path
 
-from ..base.evaluator import BaseEvaluator
+from ..base import BaseEvaluator
 from .support_models import SUPPORT_MODELS
 
 

paddlex/modules/ts_forecast/trainer.py

@@ -20,8 +20,7 @@ import tarfile
 from pathlib import Path
 import paddle
 
-from ..base.trainer import BaseTrainer
-from ..base.train_deamon import BaseTrainDeamon
+from ..base import BaseTrainer, BaseTrainDeamon
 from ...utils.config import AttrDict
 from .support_models import SUPPORT_MODELS
 

paddlex/paddlex.py → paddlex/paddlex_cli.py

@@ -19,9 +19,9 @@ import textwrap
 from types import SimpleNamespace
 from prettytable import PrettyTable
 
-from .utils.config import AttrDict
-from .modules.base.predictor import build_predictor, BasePredictor
+from .pipelines import build_pipeline, BasePipeline
 from .repo_manager import setup, get_all_supported_repo_names
+from .utils import logging
 
 
 def args_cfg():
@@ -49,30 +49,17 @@ def args_cfg():
         action='store_true',
         help="Whether to reinstall all packages.")
 
-    ################# infer #################
+    ################# pipeline predict #################
     parser.add_argument('--predict', action='store_true', default=True, help="")
-    parser.add_argument('--model_name', type=str, help="")
-    parser.add_argument('--model', type=str, help="")
-    parser.add_argument('--input_path', type=str, help="")
+    parser.add_argument('--pipeline', type=str, help="")
+    parser.add_argument('--model', nargs='+', help="")
+    parser.add_argument('--input', type=str, help="")
     parser.add_argument('--output', type=str, help="")
     parser.add_argument('--device', type=str, default='gpu:0', help="")
 
     return parser.parse_args()
 
 
-def get_all_models():
-    """Get all models that have been registered
-    """
-    all_models = BasePredictor.all()
-    model_map = {}
-    for model in all_models:
-        module = all_models[model].__name__
-        if module not in model_map:
-            model_map[module] = []
-        model_map[module].append(model)
-    return model_map
-
-
 def print_info():
     """Print list of supported models in formatted.
     """
@@ -86,26 +73,19 @@ def print_info():
         second_width = 100
     total_width -= 4
 
-    models_table = PrettyTable()
-    models_table.field_names = ["Modules", "Models"]
-    model_map = get_all_models()
-    for module in model_map:
-        models = model_map[module]
-        models_table.add_row(
-            [
-                textwrap.fill(
-                    f"{module}", width=total_width // 5), textwrap.fill(
-                        "  ".join(models), width=total_width * 4 // 5)
-            ],
-            divider=True)
+    pipeline_table = PrettyTable()
+    pipeline_table.field_names = ["Pipelines"]
+    pipeline_table.add_row(
+        [textwrap.fill(
+            ",  ".join(BasePipeline.all()), width=total_width)])
 
-    table_width = len(str(models_table).split("\n")[0])
+    table_width = len(str(pipeline_table).split("\n")[0])
 
-    print("{}".format("-" * table_width))
-    print("PaddleX".center(table_width))
-    print(models_table)
-    print("Powered by PaddlePaddle!".rjust(table_width))
-    print("{}".format("-" * table_width))
+    logging.info("{}".format("-" * table_width))
+    logging.info("PaddleX".center(table_width))
+    logging.info(pipeline_table)
+    logging.info("Powered by PaddlePaddle!".rjust(table_width))
+    logging.info("{}".format("-" * table_width))
 
 
 def install(args):
@@ -128,39 +108,9 @@ def install(args):
     return
 
 
-def build_predict_config(model_name, model, input_path, device, output):
-    """build predict config for paddlex
-    """
-
-    def dict2attrdict(dict_obj):
-        """convert dict object to AttrDict
-        """
-        for key, value in dict_obj.items():
-            if isinstance(value, dict):
-                dict_obj[key] = dict2attrdict(value)
-        return AttrDict(dict_obj)
-
-    config = {
-        'Global': {
-            'model': model_name,
-            'device': device,
-            'output': output
-        },
-        'Predict': {
-            'model_dir': model,
-            'input_path': input_path,
-        }
-    }
-
-    return dict2attrdict(config)
-
-
-def predict(model_name, model, input_path, device, output):
-    """predict using paddlex
-    """
-    config = build_predict_config(model_name, model, input_path, device, output)
-    predict = build_predictor(config)
-    return predict()
+def pipeline_predict(pipeline, model_name_list, input_path, output_dir, device):
+    pipeline = build_pipeline(pipeline, model_name_list, output_dir, device)
+    pipeline.predict(input_path)
 
 
 # for CLI
@@ -172,5 +122,5 @@ def main():
         install(args)
     else:
         print_info()
-        return predict(args.model_name, args.model, args.input_path,
-                       args.device, args.output)
+        return pipeline_predict(args.pipeline, args.model, args.input,
+                                args.output, args.device)

paddlex/pipelines/PPOCR/__init__.py

@@ -14,4 +14,4 @@
 
 
 
-from .main import OCRPipeline
+from .pipeline import OCRPipeline

paddlex/pipelines/PPOCR/main.py

@@ -1,69 +0,0 @@
-# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
-# 
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-
-from ...modules.base import create_model
-from ...modules.text_detection.predictor import transforms as text_det_T
-from .utils import draw_ocr_box_txt
-
-
-class OCRPipeline(object):
-    """OCR Pipeline
-    """
-
-    def __init__(self,
-                 text_det_model_name,
-                 text_rec_model_name,
-                 text_det_model_dir=None,
-                 text_rec_model_dir=None,
-                 text_det_kernel_option=None,
-                 text_rec_kernel_option=None,
-                 output_dir="output"):
-        self.output_dir = output_dir
-        text_det_kernel_option = self.get_kernel_option(
-        ) if text_det_kernel_option is None else text_det_kernel_option
-        text_rec_kernel_option = self.get_kernel_option(
-        ) if text_rec_kernel_option is None else text_rec_kernel_option
-
-        text_det_post_transforms = [
-            text_det_T.DBPostProcess(
-                thresh=0.3,
-                box_thresh=0.6,
-                max_candidates=1000,
-                unclip_ratio=1.5,
-                use_dilation=False,
-                score_mode='fast',
-                box_type='quad'),
-            # TODO
-            text_det_T.CropByPolys(det_box_type="foo")
-        ]
-        self.text_det_model = create_model(
-            text_det_model_name,
-            text_det_model_dir,
-            kernel_option=text_det_kernel_option,
-            post_transforms=text_det_post_transforms)
-        self.text_rec_model = create_model(
-            text_rec_model_name,
-            text_rec_model_dir,
-            kernel_option=text_rec_kernel_option)
-
-    def __call__(self, input_path):
-        result = self.text_det_model.predict({"input_path": input_path})
-        all_rec_result = []
-        for i, img in enumerate(result["sub_imgs"]):
-            rec_result = self.text_rec_model.predict({"image": img})
-            all_rec_result.append(rec_result["rec_text"][0])
-        result["rec_text"] = all_rec_result
-        return result

paddlex/pipelines/PPOCR/pipeline.py

@@ -0,0 +1,112 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import os
+import cv2
+
+from ..base import BasePipeline
+from ...modules import create_model, PaddleInferenceOption
+from ...modules.text_detection import transforms as text_det_T
+from .utils import draw_ocr_box_txt
+
+
+class OCRPipeline(BasePipeline):
+    """OCR Pipeline
+    """
+    support_models = "ocr"
+
+    def __init__(self,
+                 text_det_model_name=None,
+                 text_rec_model_name=None,
+                 text_det_model_dir=None,
+                 text_rec_model_dir=None,
+                 text_det_kernel_option=None,
+                 text_rec_kernel_option=None,
+                 output_dir=None,
+                 **kwargs):
+        self.text_det_model_name = text_det_model_name
+        self.text_rec_model_name = text_rec_model_name
+        self.text_det_model_dir = text_det_model_dir
+        self.text_rec_model_dir = text_rec_model_dir
+        self.output_dir = output_dir
+        self.text_det_kernel_option = self.get_kernel_option(
+        ) if text_det_kernel_option is None else text_det_kernel_option
+        self.text_rec_kernel_option = self.get_kernel_option(
+        ) if text_rec_kernel_option is None else text_rec_kernel_option
+
+        self.text_det_post_transforms = [
+            text_det_T.DBPostProcess(
+                thresh=0.3,
+                box_thresh=0.6,
+                max_candidates=1000,
+                unclip_ratio=1.5,
+                use_dilation=False,
+                score_mode='fast',
+                box_type='quad'),
+            # TODO
+            text_det_T.CropByPolys(det_box_type="foo")
+        ]
+        if self.text_det_model_name is not None and self.text_rec_model_name is not None:
+            self.load_model()
+
+    def load_model(self):
+        """load model predictor
+        """
+        assert self.text_det_model_name is not None and self.text_rec_model_name is not None
+        self.text_det_model = create_model(
+            self.text_det_model_name,
+            self.text_det_model_dir,
+            kernel_option=self.text_det_kernel_option,
+            post_transforms=self.text_det_post_transforms)
+        self.text_rec_model = create_model(
+            self.text_rec_model_name,
+            self.text_rec_model_dir,
+            kernel_option=self.text_rec_kernel_option)
+
+    def predict(self, input_path):
+        """predict
+        """
+        result = self.text_det_model.predict({"input_path": input_path})
+        all_rec_result = []
+        for i, img in enumerate(result["sub_imgs"]):
+            rec_result = self.text_rec_model.predict({"image": img})
+            all_rec_result.append(rec_result["rec_text"][0])
+        result["rec_text"] = all_rec_result
+
+        if self.output_dir is not None:
+            draw_img = draw_ocr_box_txt(result['original_image'],
+                                        result['dt_polys'], result["rec_text"])
+            cv2.imwrite(
+                os.path.join(self.output_dir, "ocr_result.jpg"),
+                draw_img[:, :, ::-1], )
+
+        return result
+
+    def update_model_name(self, model_name_list):
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        assert len(model_name_list) == 2
+        self.text_det_model_name = model_name_list[0]
+        self.text_rec_model_name = model_name_list[1]
+
+    def get_kernel_option(self):
+        """get kernel option
+        """
+        kernel_option = PaddleInferenceOption()
+        kernel_option.set_device("gpu")
+        return kernel_option

paddlex/pipelines/PPOCR/utils.py

@@ -101,7 +101,7 @@ def draw_box_txt_fine(img_size, box, txt, font_path=PINGFANG_FONT_FILE_PATH):
 def create_font(txt, sz, font_path=PINGFANG_FONT_FILE_PATH):
     """create font
     """
-    font_size = int(sz[1] * 0.99)
+    font_size = int(sz[1] * 0.8)
     font = ImageFont.truetype(font_path, font_size, encoding="utf-8")
     if int(PIL.__version__.split(".")[0]) < 10:
         length = font.getsize(txt)[0]

paddlex/pipelines/__init__.py

@@ -14,10 +14,11 @@
 
 
 __all__ = [
-    'OCRPipeline', 'ClsPipeline', 'DetPipeline', 'InstanceSegPipeline',
-    'SegPipeline'
+    'build_pipeline', 'BasePipeline', 'OCRPipeline', 'ClsPipeline',
+    'DetPipeline', 'InstanceSegPipeline', 'SegPipeline'
 ]
 
+from .base import build_pipeline, BasePipeline
 from .PPOCR import OCRPipeline
 from .image_classification import ClsPipeline
 from .object_detection import DetPipeline

paddlex/pipelines/base/__init__.py

@@ -0,0 +1,16 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from .pipeline import build_pipeline, BasePipeline

paddlex/pipelines/base/pipeline.py

@@ -0,0 +1,62 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from abc import ABC, abstractmethod
+
+from ...utils.misc import AutoRegisterABCMetaClass
+
+
+def build_pipeline(
+        pipeline_name: str,
+        model_list: list,
+        output_dir: str,
+        device: str, ) -> "BasePipeline":
+    """build model evaluater
+
+    Args:
+        pipeline_name (str): the pipeline name, that is name of pipeline class
+
+    Returns:
+        BasePipeline: the pipeline, which is subclass of BasePipeline.
+    """
+    pipeline = BasePipeline.get(pipeline_name)(output_dir=output_dir,
+                                               device=device)
+    pipeline.update_model_name(model_list)
+    pipeline.load_model()
+    return pipeline
+
+
+class BasePipeline(ABC, metaclass=AutoRegisterABCMetaClass):
+    """Base Pipeline
+    """
+    __is_base = True
+
+    def __init__(self):
+        super().__init__()
+
+    @abstractmethod
+    def load_model(self):
+        """load model predictor
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def update_model_name(self, model_list: list) -> dict:
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        raise NotImplementedError

paddlex/pipelines/image_classification/__init__.py

@@ -14,4 +14,4 @@
 
 
 
-from .main import ClsPipeline
+from .pipeline import ClsPipeline

paddlex/pipelines/image_classification/main.py

@@ -1,54 +0,0 @@
-# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
-# 
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-
-from ...modules.base import create_model
-from ...modules.image_classification.predictor import transforms as T
-from ...modules.base.predictor.utils.paddle_inference_predictor import PaddleInferenceOption
-
-
-class ClsPipeline(object):
-    """Cls Pipeline
-    """
-
-    def __init__(self,
-                 model_name,
-                 model_dir=None,
-                 output_dir="./output",
-                 kernel_option=None):
-        self.output_dir = output_dir
-        post_transforms = self.get_post_transforms(model_dir)
-        kernel_option = self.get_kernel_option(
-        ) if kernel_option is None else kernel_option
-
-        self.model = create_model(
-            model_name,
-            model_dir=model_dir,
-            kernel_option=kernel_option,
-            post_transforms=post_transforms)
-
-    def __call__(self, input_path):
-        return self.model.predict({"input_path": input_path})
-
-    def get_post_transforms(self, model_dir):
-        """get post transform ops
-        """
-        return [T.Topk(topk=1), T.PrintResult()]
-
-    def get_kernel_option(self):
-        """get kernel option
-        """
-        kernel_option = PaddleInferenceOption()
-        kernel_option.set_device("gpu")

paddlex/pipelines/image_classification/pipeline.py

@@ -0,0 +1,74 @@
+# copyright (c) 2024 PaddlePaddle Authors. All Rights Reserve.
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from ..base import BasePipeline
+from ...modules import create_model, PaddleInferenceOption
+from ...modules.image_classification import transforms as T
+
+
+class ClsPipeline(BasePipeline):
+    """Cls Pipeline
+    """
+    support_models = "image_classification"
+
+    def __init__(self,
+                 model_name=None,
+                 model_dir=None,
+                 kernel_option=None,
+                 **kwargs):
+        super().__init__()
+        self.model_name = model_name
+        self.model_dir = model_dir
+        self.kernel_option = self.get_kernel_option(
+        ) if kernel_option is None else kernel_option
+        self.post_transforms = self.get_post_transforms()
+        if self.model_name is not None:
+            self.load_model()
+
+    def predict(self, input_path):
+        """predict
+        """
+        return self.model.predict({"input_path": input_path})
+
+    def load_model(self):
+        """load model predictor
+        """
+        assert self.model_name is not None
+        self.model = create_model(
+            self.model_name,
+            model_dir=self.model_dir,
+            kernel_option=self.kernel_option,
+            post_transforms=self.post_transforms)
+
+    def get_post_transforms(self):
+        """get post transform ops
+        """
+        return [T.Topk(topk=1), T.PrintResult()]
+
+    def get_kernel_option(self):
+        """get kernel option
+        """
+        kernel_option = PaddleInferenceOption()
+        kernel_option.set_device("gpu")
+        return kernel_option
+
+    def update_model_name(self, model_name_list):
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        assert len(model_name_list) == 1
+        self.model_name = model_name_list[0]

paddlex/pipelines/instance_segmentation/__init__.py

@@ -14,4 +14,4 @@
 
 
 
-from .main import InstanceSegPipeline
+from .pipeline import InstanceSegPipeline

paddlex/pipelines/instance_segmentation/main.py → paddlex/pipelines/instance_segmentation/pipeline.py

@@ -13,32 +13,44 @@
 # limitations under the License.
 
 
-from ...modules.base import create_model
-from ...modules.object_detection.predictor import transforms as T
-from ...modules.base.predictor.utils.paddle_inference_predictor import PaddleInferenceOption
+from ..base import BasePipeline
+from ...modules import create_model, PaddleInferenceOption
+from ...modules.object_detection import transforms as T
 
 
-class InstanceSegPipeline(object):
+class InstanceSegPipeline(BasePipeline):
     """InstanceSeg Pipeline
     """
+    support_models = "instance_segmentation"
 
     def __init__(self,
-                 model_name,
+                 model_name=None,
                  model_dir=None,
                  output_dir="./output",
-                 kernel_option=None):
+                 kernel_option=None,
+                 **kwargs):
+        self.model_name = model_name
+        self.model_dir = model_dir
         self.output_dir = output_dir
-        post_transforms = self.get_post_transforms(model_dir)
-        kernel_option = self.get_kernel_option(
+        self.post_transforms = self.get_post_transforms(model_dir)
+        self.kernel_option = self.get_kernel_option(
         ) if kernel_option is None else kernel_option
+        if self.model_name is not None:
+            self.load_model()
 
+    def load_model(self):
+        """load model predictor
+        """
+        assert self.model_name is not None
         self.model = create_model(
-            model_name,
-            model_dir=model_dir,
-            kernel_option=kernel_option,
-            post_transforms=post_transforms)
+            self.model_name,
+            model_dir=self.model_dir,
+            kernel_option=self.kernel_option,
+            post_transforms=self.post_transforms)
 
-    def __call__(self, input_path):
+    def predict(self, input_path):
+        """predict
+        """
         return self.model.predict({"input_path": input_path})
 
     def get_post_transforms(self, model_dir):
@@ -51,3 +63,13 @@ class InstanceSegPipeline(object):
         """
         kernel_option = PaddleInferenceOption()
         kernel_option.set_device("gpu")
+        return kernel_option
+
+    def update_model_name(self, model_name_list):
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        assert len(model_name_list) == 1
+        self.model_name = model_name_list[0]

paddlex/pipelines/object_detection/__init__.py

@@ -14,4 +14,4 @@
 
 
 
-from .main import DetPipeline
+from .pipeline import DetPipeline

paddlex/pipelines/object_detection/main.py → paddlex/pipelines/object_detection/pipeline.py

@@ -13,33 +13,44 @@
 # limitations under the License.
 
 
+from ..base import BasePipeline
+from ...modules import create_model, PaddleInferenceOption
+from ...modules.object_detection import transforms as T
 
-from ...modules.base import create_model
-from ...modules.object_detection.predictor import transforms as T
-from ...modules.base.predictor.utils.paddle_inference_predictor import PaddleInferenceOption
 
-
-class DetPipeline(object):
+class DetPipeline(BasePipeline):
     """Det Pipeline
     """
+    support_models = "object_detection"
 
     def __init__(self,
-                 model_name,
+                 model_name=None,
                  model_dir=None,
                  output_dir="./output",
-                 kernel_option=None):
+                 kernel_option=None,
+                 **kwargs):
+        self.model_name = model_name
+        self.model_dir = model_dir
         self.output_dir = output_dir
-        post_transforms = self.get_post_transforms(model_dir)
-        kernel_option = self.get_kernel_option(
+        self.post_transforms = self.get_post_transforms(model_dir)
+        self.kernel_option = self.get_kernel_option(
         ) if kernel_option is None else kernel_option
+        if self.model_name is not None:
+            self.load_model()
 
+    def load_model(self):
+        """load model predictor
+        """
+        assert self.model_name is not None
         self.model = create_model(
-            model_name,
-            model_dir=model_dir,
-            kernel_option=kernel_option,
-            post_transforms=post_transforms)
+            self.model_name,
+            model_dir=self.model_dir,
+            kernel_option=self.kernel_option,
+            post_transforms=self.post_transforms)
 
-    def __call__(self, input_path):
+    def predict(self, input_path):
+        """predict
+        """
         return self.model.predict({"input_path": input_path})
 
     def get_post_transforms(self, model_dir):
@@ -52,3 +63,12 @@ class DetPipeline(object):
         """
         kernel_option = PaddleInferenceOption()
         kernel_option.set_device("gpu")
+
+    def update_model_name(self, model_name_list):
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        assert len(model_name_list) == 1
+        self.model_name = model_name_list[0]

paddlex/pipelines/semantic_segmentation/__init__.py

@@ -14,4 +14,4 @@
 
 
 
-from .main import SegPipeline
+from .pipeline import SegPipeline

paddlex/pipelines/semantic_segmentation/main.py → paddlex/pipelines/semantic_segmentation/pipeline.py

@@ -13,33 +13,44 @@
 # limitations under the License.
 
 
+from ..base import BasePipeline
+from ...modules import create_model, PaddleInferenceOption
+from ...modules.semantic_segmentation import transforms as T
 
-from ...modules.base import create_model
-from ...modules.semantic_segmentation.predictor import transforms as T
-from ...modules.base.predictor.utils.paddle_inference_predictor import PaddleInferenceOption
 
-
-class SegPipeline(object):
+class SegPipeline(BasePipeline):
     """Seg Pipeline
     """
+    support_models = "semantic_segmentation"
 
     def __init__(self,
-                 model_name,
+                 model_name=None,
                  model_dir=None,
                  output_dir="./output",
-                 kernel_option=None):
+                 kernel_option=None,
+                 **kwargs):
+        self.model_name = model_name
+        self.model_dir = model_dir
         self.output_dir = output_dir
-        post_transforms = self.get_post_transforms()
-        kernel_option = self.get_kernel_option(
+        self.post_transforms = self.get_post_transforms()
+        self.kernel_option = self.get_kernel_option(
         ) if kernel_option is None else kernel_option
+        if self.model_name is not None:
+            self.load_model()
 
+    def load_model(self):
+        """load model predictor
+        """
+        assert self.model_name is not None
         self.model = create_model(
-            model_name,
-            model_dir=model_dir,
-            kernel_option=kernel_option,
-            post_transforms=post_transforms)
+            self.model_name,
+            model_dir=self.model_dir,
+            kernel_option=self.kernel_option,
+            post_transforms=self.post_transforms)
 
-    def __call__(self, input_path):
+    def predict(self, input_path):
+        """predict
+        """
         return self.model.predict({"input_path": input_path})
 
     def get_post_transforms(self):
@@ -52,3 +63,13 @@ class SegPipeline(object):
         """
         kernel_option = PaddleInferenceOption()
         kernel_option.set_device("gpu")
+        return kernel_option
+
+    def update_model_name(self, model_name_list):
+        """update model name and re
+
+        Args:
+            model_list (list): list of model name.
+        """
+        assert len(model_name_list) == 1
+        self.model_name = model_name_list[0]

paddlex/repo_manager/meta.py

@@ -16,10 +16,6 @@
 
 __all__ = ['get_all_repo_names']
 
-# REPO_NAMES = [
-#     'PaddleSeg', 'Paddle3D', 'PaddleClas', 'PaddleDetection', 'PaddleOCR',
-#     'PaddleTS', 'PaddleNLP', 'PaddleSpeech', 'PARL', 'PaddleMIX']
-
 REPO_NAMES = [
     'PaddleClas', 'PaddleOCR', 'PaddleDetection', 'PaddleSeg', 'PaddleNLP',
     'PaddleTS'
@@ -28,7 +24,8 @@ REPO_NAMES = [
 REPO_META = {
     'PaddleSeg': {
         'repo_url': '/PaddlePaddle/PaddleSeg.git',
-        'branch': 'develop',
+        'platform': 'github',
+        'branch': 'release/2.9.1',
         'pkg_name': 'paddleseg',
         'lib_name': 'paddleseg',
         'pdx_pkg_name': 'PaddleSeg_api',
@@ -36,20 +33,10 @@ REPO_META = {
         'extra_req_files': ['Matting/requirements.txt'],
         'path_env': 'PADDLE_PDX_PADDLESEG_PATH',
     },
-    'Paddle3D': {
-        'repo_url': '/PaddlePaddle/Paddle3D.git',
-        'branch': 'develop',
-        'pkg_name': 'paddle3d',
-        'lib_name': 'paddle3d',
-        'pdx_pkg_name': 'Paddle3D_api',
-        'editable': False,
-        'path_env': 'PADDLE_PDX_PADDLE3D_PATH',
-        'requires': ['PaddleSeg', 'PaddleDetection'],
-        'pdx_pkg_deps': ['nuscenes-devkit', 'pyquaternion'],
-    },
     'PaddleClas': {
         'repo_url': '/PaddlePaddle/PaddleClas.git',
-        'branch': 'develop',
+        'platform': 'github',
+        'branch': 'release/2.5.2',
         'pkg_name': 'paddleclas',
         'lib_name': 'paddleclas',
         'pdx_pkg_name': 'PaddleClas_api',
@@ -60,7 +47,8 @@ REPO_META = {
     },
     'PaddleDetection': {
         'repo_url': '/PaddlePaddle/PaddleDetection.git',
-        'branch': 'develop',
+        'platform': 'github',
+        'branch': 'release/2.7.1',
         'pkg_name': 'paddledet',
         'lib_name': 'ppdet',
         'pdx_pkg_name': 'PaddleDetection_api',
@@ -81,6 +69,7 @@ REPO_META = {
     },
     'PaddleTS': {
         'repo_url': '/PaddlePaddle/PaddleTS.git',
+        'platform': 'github',
         'branch': 'release_v1.1',
         'pkg_name': 'paddlets',
         'lib_name': 'paddlets',
@@ -91,6 +80,7 @@ REPO_META = {
     },
     'PaddleNLP': {
         'repo_url': '/PaddlePaddle/PaddleNLP.git',
+        'platform': 'github',
         'branch': 'develop',
         'pkg_name': 'paddlenlp',
         'lib_name': 'paddlenlp',
@@ -100,6 +90,7 @@ REPO_META = {
     },
     'PaddleSpeech': {
         'repo_url': '/PaddlePaddle/PaddleSpeech.git',
+        'platform': 'github',
         'branch': 'develop',
         'pkg_name': 'paddlespeech',
         'lib_name': 'paddlespeech',
@@ -110,6 +101,7 @@ REPO_META = {
     },
     'PARL': {
         'repo_url': '/PaddlePaddle/PARL.git',
+        'platform': 'github',
         'branch': 'develop',
         'pkg_name': 'parl',
         'lib_name': 'parl',
@@ -119,6 +111,7 @@ REPO_META = {
     },
     'PaddleMIX': {
         'repo_url': '/PaddlePaddle/PaddleMIX.git',
+        'platform': 'github',
         'branch': 'develop',
         'pkg_name': 'paddlemix',
         'lib_name': 'paddlemix',

paddlex/utils/file_interface.py

@@ -98,6 +98,9 @@ def custom_open(file_path, mode):
             mode))
 
 
+# --------------- yaml ---------------
+
+
 def read_yaml_file(yaml_path: str, to_dict=True):
     """read from yaml file"""
     try:

paddlex/utils/logging.py

@@ -76,8 +76,12 @@ def setup_logging(verbosity: str=None):
     Args:
         verbosity (str, optional): the logging level, `DEBUG`, `INFO`, `WARNING`. Defaults to None.
     """
-    if verbosity is None and DEBUG:
-        verbosity = 'DEBUG'
+    if verbosity is None:
+        if DEBUG:
+            verbosity = 'DEBUG'
+        else:
+            verbosity = 'INFO'
+
     if verbosity is not None:
         _configure_logger(_logger, verbosity.upper())
 

requirements.txt

@@ -12,4 +12,5 @@ tqdm
 pyclipper
 shapely
 pandas
-parsley
+parsley
+requests

setup.py

@@ -60,6 +60,8 @@ def packages_and_package_data():
         'paddlex',
         'paddlex.modules',
         'paddlex.modules.*',
+        'paddlex.pipelines',
+        'paddlex.pipelines.*',
         'paddlex.repo_manager',
         'paddlex.repo_apis',
         'paddlex.repo_apis.*',
@@ -80,8 +82,8 @@ def packages_and_package_data():
         rp = '/'.join(parts[1:])
         pkg_data.append(rp)
     pkg_data.append('.version')
-    pkg_data.append('paddlex/modules/utils/fonts/PingFang-SC-Regular.ttf')
-    pkg_data.append('paddlex/repo_manager/requirements.txt')
+    pkg_data.append('utils/fonts/PingFang-SC-Regular.ttf')
+    pkg_data.append('repo_manager/requirements.txt')
     return pkgs, {'paddlex': pkg_data}
 
 
@@ -89,7 +91,7 @@ def check_paddle_version():
     """check paddle version
     """
     import paddle
-    supported_versions = ['2.5', '2.6', '0.0']
+    supported_versions = ['2.6', '3.0', '0.0']
     version = paddle.__version__
     # Recognizable version number: major.minor.patch
     major, minor, patch = version.split('.')
@@ -110,15 +112,15 @@ if __name__ == '__main__':
     s = setup(
         name='paddlex',
         version=version(),
-        description=('PaddlePaddle ultra API.'),
+        description=('Low-code development tool based on PaddlePaddle.'),
         long_description=readme(),
         author='PaddlePaddle Authors',
         author_email='',
-        # install_requires=dependencies(),
+        install_requires=dependencies(),
         packages=pkgs,
         package_data=pkg_data,
         entry_points={
-            'console_scripts': ['paddlex = paddlex.paddlex:main', ],
+            'console_scripts': ['paddlex = paddlex.paddlex_cli:main', ],
         },
         # PyPI package information
         classifiers=[
@@ -127,8 +129,9 @@ if __name__ == '__main__':
             'Intended Audience :: Education',
             'Intended Audience :: Science/Research',
             'License :: OSI Approved :: Apache Software License',
-            'Programming Language :: Python :: 3.7',
             'Programming Language :: Python :: 3.8',
+            'Programming Language :: Python :: 3.9',
+            'Programming Language :: Python :: 3.10',
             'Topic :: Scientific/Engineering',
             'Topic :: Scientific/Engineering :: Mathematics',
             'Topic :: Scientific/Engineering :: Artificial Intelligence',