MinerU/next_docs/zh_cn/user_guide/tutorial/pipeline.rst

流水线管道
===========


极简示例
^^^^^^^^

.. code:: python

    import os

    from magic_pdf.data.data_reader_writer import FileBasedDataWriter, FileBasedDataReader
    from magic_pdf.data.dataset import PymuDocDataset
    from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze

    # args
    pdf_file_name = "abc.pdf"  # replace with the real pdf path
    name_without_suff = pdf_file_name.split(".")[0]

    # prepare env
    local_image_dir, local_md_dir = "output/images", "output"
    image_dir = str(os.path.basename(local_image_dir))

    os.makedirs(local_image_dir, exist_ok=True)

    image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(
        local_md_dir
    )
    image_dir = str(os.path.basename(local_image_dir))

    # read bytes
    reader1 = FileBasedDataReader("")
    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content

    # proc
    ## Create Dataset Instance
    ds = PymuDocDataset(pdf_bytes)

    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)


运行以上的代码，会得到如下的结果

.. code:: bash 

    output/
    ├── abc.md
    └── images


除去初始化环境，如建立目录、导入依赖库等逻辑。真正将 ``pdf`` 转换为 ``markdown`` 的代码片段如下

.. code::

    # read bytes
    reader1 = FileBasedDataReader("")
    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content

    # proc
    ## Create Dataset Instance
    ds = PymuDocDataset(pdf_bytes)

    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)


``ds.apply(doc_analyze, ocr=True)`` 会生成 ``InferenceResult`` 对象。 ``InferenceResult`` 对象执行 ``pipe_ocr_mode`` 方法会生成 ``PipeResult`` 对象。
``PipeResult`` 对象执行 ``dump_md`` 会在指定位置生成 ``markdown`` 文件。


pipeline 的执行过程如下图所示

.. image:: ../../_static/image/pipeline.drawio.svg 

.. raw:: html 

    <br> </br>

目前划分出数据、推理、程序处理三个阶段，分别对应着图上的 ``Dataset``， ``InferenceResult``， ``PipeResult`` 这三个实体。通过 ``apply`` ， ``doc_analyze`` 或 ``pipe_ocr_mode`` 等方法链接在一起。


.. admonition:: Tip
    :class: tip

    要想获得更多有关 Dataset、InferenceResult、PipeResult 的使用示例子，请前往 :doc:`../quick_start/to_markdown`

    要想获得更多有关 Dataset、InferenceResult、PipeResult 的细节信息请前往英文版 MinerU 文档进行查看!



管道组合
^^^^^^^^^

.. code:: python

    class Dataset(ABC):
        @abstractmethod
        def apply(self, proc: Callable, *args, **kwargs):
            """Apply callable method which.

            Args:
                proc (Callable): invoke proc as follows:
                    proc(self, *args, **kwargs)

            Returns:
                Any: return the result generated by proc
            """
            pass

    class InferenceResult(InferenceResultBase):

        def apply(self, proc: Callable, *args, **kwargs):
            """Apply callable method which.

            Args:
                proc (Callable): invoke proc as follows:
                    proc(inference_result, *args, **kwargs)

            Returns:
                Any: return the result generated by proc
            """
            return proc(copy.deepcopy(self._infer_res), *args, **kwargs)

        def pipe_ocr_mode(
            self,
            imageWriter: DataWriter,
            start_page_id=0,
            end_page_id=None,
            debug_mode=False,
            lang=None,
            ) -> PipeResult:
            pass

    class PipeResult:
        def apply(self, proc: Callable, *args, **kwargs):
            """Apply callable method which.

            Args:
                proc (Callable): invoke proc as follows:
                    proc(pipeline_result, *args, **kwargs)

            Returns:
                Any: return the result generated by proc
            """
            return proc(copy.deepcopy(self._pipe_res), *args, **kwargs)

``Dataset`` 、 ``InferenceResult`` 和 ``PipeResult`` 类均有 ``apply`` method。可用于组合不同阶段的运算过程。
如下所示，``MinerU`` 提供一套组合这些类的计算过程。

.. code:: python 

    # proc
    ## Create Dataset Instance
    ds = PymuDocDataset(pdf_bytes)

    ds.apply(doc_analyze, ocr=True).pipe_ocr_mode(image_writer).dump_md(md_writer, f"{name_without_suff}.md", image_dir)

用户可以根据的需求，自行实现一些组合用的函数。比如用户通过 ``apply`` 方法实现一个统计 ``pdf`` 文件页数的功能。

.. code:: python 

    from magic_pdf.data.data_reader_writer import  FileBasedDataReader
    from magic_pdf.data.dataset import PymuDocDataset

    # args
    pdf_file_name = "abc.pdf"  # replace with the real pdf path

    # read bytes
    reader1 = FileBasedDataReader("")
    pdf_bytes = reader1.read(pdf_file_name)  # read the pdf content

    # proc
    ## Create Dataset Instance
    ds = PymuDocDataset(pdf_bytes)

    def count_page(ds)-> int:
        return len(ds)

    print("page number: ", ds.apply(count_page)) # will output the page count of `abc.pdf`