feat: add zh_cn docs

magic_pdf/data/dataset.py

@@ -109,7 +109,7 @@ class Dataset(ABC):
 
         Args:
             proc (Callable): invoke proc as follows:
-                proc(dataset, *args, **kwargs)
+                proc(self, *args, **kwargs)
 
         Returns:
             Any: return the result generated by proc

next_docs/zh_cn/user_guide/tutorial.rst

@@ -9,3 +9,5 @@
     :caption: 教程
 
     tutorial/output_file_description
+    tutorial/pipeline
+

next_docs/zh_cn/user_guide/tutorial/pipeline.rst

@@ -0,0 +1,175 @@
+
+流水线管道
+===========
+
+
+极简示例
+^^^^^^^^
+
+.. code:: python
+
+    import os
+
+    from magic_pdf.data.data_reader_writer import FileBasedDataWriter, FileBasedDataReader
+    from magic_pdf.data.dataset import PymuDocDataset
+    from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
+
+    # args
+    pdf_file_name = "abc.pdf"  # replace with the real pdf path
+    name_without_suff = pdf_file_name.split(".")[0]
+
+    # prepare env
+    local_image_dir, local_md_dir = "output/images", "output"
+    image_dir = str(os.path.basename(local_image_dir))
+
+    os.makedirs(local_image_dir, exist_ok=True)
+
+    image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(
+        local_md_dir
+    )
+    image_dir = str(os.path.basename(local_image_dir))
+
+    # read bytes
+    reader1 = FileBasedDataReader("")
+    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content
+
+    # proc
+    ## Create Dataset Instance
+    ds = PymuDocDataset(pdf_bytes)
+
+    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)
+
+
+运行以上的代码，会得到如下的结果
+
+.. code:: bash 
+
+    output/
+    ├── abc.md
+    └── images
+
+
+除去初始化环境，如建立目录、导入依赖库等逻辑。真正执行将 `pdf` 转换为 `markdown` 的代码片段如下
+
+.. code::
+
+    # read bytes
+    reader1 = FileBasedDataReader("")
+    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content
+
+    # proc
+    ## Create Dataset Instance
+    ds = PymuDocDataset(pdf_bytes)
+
+    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)
+
+
+`ds.apply(doc_analyze, ocr=True)` 会生成 `InferenceResult` 对象。 `InferenceResult` 的对象执行 `pipe_ocr_mode` 方法会生成 `PipeResult` 对象。
+`PipeResult` 对象执行 `dump_md` 会在指定位置生成 `markdown` 文件。
+
+
+pipeline 的执行过程如下图所示
+
+.. image:: ../../_static/image/pipeline.drawio.svg 
+
+
+
+
+.. admonition:: Tip
+    :class: tip
+
+    要想获得更多有关 Dataset、InferenceResult、PipeResult 的使用示例子，请前往 :doc:`../quick_start/to_markdown`
+
+    要想获得更多有关 Dataset、InferenceResult、PipeResult 的细节信息请前往英文版 MinerU 文档进行查看!
+
+
+
+管道组合
+^^^^^^^^^
+
+.. code:: python
+
+    class Dataset(ABC):
+        @abstractmethod
+        def apply(self, proc: Callable, *args, **kwargs):
+            """Apply callable method which.
+
+            Args:
+                proc (Callable): invoke proc as follows:
+                    proc(self, *args, **kwargs)
+
+            Returns:
+                Any: return the result generated by proc
+            """
+            pass
+
+    class InferenceResult(InferenceResultBase):
+
+        def apply(self, proc: Callable, *args, **kwargs):
+            """Apply callable method which.
+
+            Args:
+                proc (Callable): invoke proc as follows:
+                    proc(inference_result, *args, **kwargs)
+
+            Returns:
+                Any: return the result generated by proc
+            """
+            return proc(copy.deepcopy(self._infer_res), *args, **kwargs)
+
+        def pipe_ocr_mode(
+            self,
+            imageWriter: DataWriter,
+            start_page_id=0,
+            end_page_id=None,
+            debug_mode=False,
+            lang=None,
+            ) -> PipeResult:
+            pass
+
+    class PipeResult:
+        def apply(self, proc: Callable, *args, **kwargs):
+            """Apply callable method which.
+
+            Args:
+                proc (Callable): invoke proc as follows:
+                    proc(pipeline_result, *args, **kwargs)
+
+            Returns:
+                Any: return the result generated by proc
+            """
+            return proc(copy.deepcopy(self._pipe_res), *args, **kwargs)
+
+`Dataset` 、 `InferenceResult` 和 `PipeResult` 类均有 `apply` method。可用于组合不同阶段的运算过程。
+如下所示，`MinerU` 提供一套组合这些类的计算过程。
+
+.. code:: python 
+
+    # proc
+    ## Create Dataset Instance
+    ds = PymuDocDataset(pdf_bytes)
+
+    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)
+
+用户可以根据的需求，自行实现一些组合用的函数。比如用户通过 `apply` 方法实现一个统计 `pdf` 文件页数的功能。
+
+.. code:: python 
+
+    from magic_pdf.data.data_reader_writer import  FileBasedDataReader
+    from magic_pdf.data.dataset import PymuDocDataset
+
+    # args
+    pdf_file_name = "abc.pdf"  # replace with the real pdf path
+
+    # read bytes
+    reader1 = FileBasedDataReader("")
+    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content
+
+    # proc
+    ## Create Dataset Instance
+    ds = PymuDocDataset(pdf_bytes)
+
+    def count_page(ds)-> int:
+        return len(ds)
+
+    print("page number: ", ds.apply(count_page)) # will output the page count of `abc.pdf`