Script for pipeline with structure recognition using table-transformer available here #116

JaMe76 · 2023-02-09T09:31:52Z

JaMe76
Feb 9, 2023
Maintainer

With the latest PR, it is now possible to perform table recognition and table structure recognition with MS table-transformer, as suggested in repo https://github.com/microsoft/table-transformer. I use, as so often, the integration of the model via the Transformer library.

I am very pleased with the results, especially regarding row and column recognition: it seems that the model generalizes better when adapted to a different domain than medical research tables compared to the current model trained on Pubtabnet.
With regard to spanning cells, column_header, etc. detection, I am more cautious. There might also be a problem when generating the HTML which can be due to poor spanning cell prediction but I haven't figured out yet.
There is of course the possibility to filter spanning cells and generate a table structure only with simple cells cells. But I am not quite certain about that.

However, to integrate the model into a robust processing pipeline, several configurations need to be set (padding, nms thresholds, ...). Attached is a proposal of a pipeline with a given config, but I am interested if there are better options. You can copy/paste this script and get started. Feedback and suggestions on this are always welcome.

table-transformer will be available in the next release - I'll try to create one by mid-February. If this approach gets more robust I might consider replacing the built-in pipeline with this one, but not at this stage.

from typing import List

from deepdoctection.utils import LayoutType, get_configs_dir_path, CellType
from deepdoctection.utils.metacfg import AttrDict
from deepdoctection.pipe import (
    PipelineComponent,
    ImageLayoutService,
    SubImageLayoutService,
    PubtablesSegmentationService,
    TextExtractionService,
    TextOrderService,
    MatchingService,
    DoctectionPipe,
    DetectResultGenerator,
)

from deepdoctection.extern import ModelCatalog, ModelDownloadManager, D2FrcnnDetector, HFDetrDerivedDetector, TesseractOcrDetector, ModelProfile
from deepdoctection.utils import set_logger_dir
from deepdoctection.utils import PadTransform
import deepdoctection as dd

def build_detr_analyzer(cfg: AttrDict):
    pipe_component_list: List[PipelineComponent] = []

    layout_config_path = ModelCatalog.get_full_path_configs(cfg.CONFIG.LAYOUT)
    layout_weights_path = ModelDownloadManager.maybe_download_weights_and_configs(cfg.WEIGHTS.LAYOUT)
    profile_layout = ModelCatalog.get_profile(cfg.WEIGHTS.LAYOUT)
    categories_layout = profile_layout.categories
    assert categories_layout is not None
    assert layout_weights_path is not None
    d_layout = D2FrcnnDetector(layout_config_path, layout_weights_path, categories_layout, device=cfg.DEVICE)
    layout = ImageLayoutService(d_layout, to_image=True, crop_image=True)
    pipe_component_list.append(layout)

    structure_config_path = ModelCatalog.get_full_path_configs(cfg.CONFIG.STRUCTURE)
    structure_weights_path = ModelDownloadManager.maybe_download_weights_and_configs(cfg.WEIGHTS.STRUCTURE)
    structure_preprocessor_config = ModelCatalog.get_full_path_preprocessor_configs(cfg.CONFIG.STRUCTURE)
    profile = ModelCatalog.get_profile(cfg.WEIGHTS.STRUCTURE)
    categories_structure = profile.categories
    hf_table_structure = HFDetrDerivedDetector(structure_config_path,
                                            structure_weights_path,
                                            structure_preprocessor_config,
                                            categories_structure,
                                            filter_categories=[LayoutType.table])
    padder = PadTransform(top=cfg.PAD.TOP,right=cfg.PAD.RIGHT,bottom=cfg.PAD.BOTTOM,left=cfg.PAD.LEFT)
    detect_result_generator = DetectResultGenerator(categories_structure, exclude_category_ids=["1","3","4","5","6"])
    structure = SubImageLayoutService(hf_table_structure,
                                      LayoutType.table,
                                      {1: 6, 2:7, 3:8, 4:9, 5:10, 6:11},
                                      detect_result_generator,
                                      padder)
    pipe_component_list.append(structure)
    pubtables = PubtablesSegmentationService(cfg.SEGMENTATION.ASSIGNMENT_RULE,
                                             cfg.SEGMENTATION.THRESHOLD_ROWS,
                                             cfg.SEGMENTATION.THRESHOLD_COLS,
                                             cfg.SEGMENTATION.FULL_TABLE_TILING,
                                             cfg.SEGMENTATION.REMOVE_IOU_THRESHOLD_ROWS,
                                             cfg.SEGMENTATION.REMOVE_IOU_THRESHOLD_COLS,
                                             cfg.SEGMENTATION.CELL_CATEGORY_ID,
                                             stretch_rule=cfg.SEGMENTATION.STRETCH_RULE)
    pipe_component_list.append(pubtables)

    tess_ocr_config_path = get_configs_dir_path() / cfg.CONFIG.TESS_OCR
    d_tess_ocr = TesseractOcrDetector(
    tess_ocr_config_path, config_overwrite=[f"LANGUAGES={cfg.LANG}"] if cfg.LANG is not None else None
)
    text = TextExtractionService(d_tess_ocr)
    pipe_component_list.append(text)

    match = MatchingService(
        parent_categories=cfg.WORD_MATCHING.PARENTAL_CATEGORIES,
        child_categories=LayoutType.word,
        matching_rule=cfg.WORD_MATCHING.RULE,
        threshold=cfg.WORD_MATCHING.THRESHOLD,
        max_parent_only=cfg.WORD_MATCHING.MAX_PARENT_ONLY
    )
    pipe_component_list.append(match)

    order = TextOrderService(
        text_container=LayoutType.word,
        floating_text_block_names=[LayoutType.title, LayoutType.text, LayoutType.list],
        text_block_names=[
            LayoutType.title,
            LayoutType.text,
            LayoutType.list,
            LayoutType.cell,
            CellType.spanning,
            CellType.projected_row_header,
            CellType.column_header,
            CellType.row_header,
        ],
    )
    pipe_component_list.append(order)

    pipe = DoctectionPipe(pipeline_component_list=pipe_component_list)
    return pipe

def set_cfg():
    config_dict = {"CONFIG" : {"LAYOUT": "dd/d2/layout/CASCADE_RCNN_R_50_FPN_GN.yaml",
                               "STRUCTURE": "microsoft/table-transformer-structure-recognition/pytorch_model.bin",
                               "TESS_OCR": "dd/conf_tesseract.yaml"
                               },
                   "PAD" : {"TOP": 60, "RIGHT": 60, "BOTTOM": 60, "LEFT": 60},
                   "WEIGHTS": {"LAYOUT" : "layout/d2_model_0829999_layout_inf_only.pt",
                               "STRUCTURE": "microsoft/table-transformer-structure-recognition/pytorch_model.bin"
                               },
                   "SEGMENTATION": {"ASSIGNMENT_RULE": "ioa",
                   "THRESHOLD_ROWS": 0.4,
                   "THRESHOLD_COLS": 0.4,
                   "FULL_TABLE_TILING": True,
    "REMOVE_IOU_THRESHOLD_ROWS": 0.9,
    "REMOVE_IOU_THRESHOLD_COLS": 0.9,
    "CELL_CATEGORY_ID": 12,
    "STRETCH_RULE": "equal",
    },
    "LANG": "eng",
    "WORD_MATCHING": {"PARENTAL_CATEGORIES": ["text", "title", "cell","list","column_header","projected_row_header","spanning"],
                      "RULE": "ioa",
                      "THRESHOLD": 0.6,
                      "MAX_PARENT_ONLY": True,
                      },
    "DEVICE": None,
                   }
    cfg = AttrDict()
    cfg.from_dict(config_dict)
    return cfg


if __name__=="__main__":

    set_logger_dir("path/to/logs")   # set logger file to get insights from warnings
    cfg = set_cfg()
    analyzer = build_detr_analyzer(cfg)

    path = "path/to/dir"

    df = analyzer.analyze(path=path, output="image")
    df.reset_state()

    for idx, dp in enumerate(df):
        page = dd.Page.from_image(dp,text_container=dd.LayoutType.word,top_level_text_block_names=[layout for layout in dd.LayoutType if layout!=dd.LayoutType.cell])
        page.viz(interactive=True)

jayeshthukarul · 2023-02-09T10:15:28Z

jayeshthukarul
Feb 9, 2023

Thanks.

0 replies

buntys2010 · 2023-02-28T14:19:16Z

buntys2010
Feb 28, 2023

Thank you for amazing repo, this makes complicated Document AI process much simpler.

When i am trying above code its giving me below error

`Downloading (…)lve/main/config.json: 100%
1.47k/1.47k [00:00<00:00, 71.9kB/s]
You are using a model of type table-transformer to instantiate a model of type . This is not supported for all configurations of models and can yield errors.

AttributeError Traceback (most recent call last)
in
131 set_logger_dir("path/to/logs") # set logger file to get insights from warnings
132 cfg = set_cfg()
--> 133 analyzer = build_detr_analyzer(cfg)
134
135 path = "path/to/dir"

7 frames
/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py in getattribute(self, key)
258 if key != "attribute_map" and key in super().getattribute("attribute_map"):
259 key = super().getattribute("attribute_map")[key]
--> 260 return super().getattribute(key)
261
262 def init(self, **kwargs):

AttributeError: 'PretrainedConfig' object has no attribute 'use_timm_backbone'
Downloading (…)rocessor_config.json: 100%
274/274 [00:00<00:00, 13.9kB/s]`

Let me know if i have missed anything here.

2 replies

JaMe76 Feb 28, 2023
Maintainer Author

I suspect this has something to do with version issues of the transformers library. I had no issues with transformers==4.24 but this might change for higher versions.

1.) Can you check with pip list what transformers version has been installed in your setup?
2.) Can you downgrade transformers to e.g. 4.24 and retry again?

If this is the root of the issue I will setup a version requirement for transformers.

buntys2010 Mar 1, 2023

Yes downgrading transformers works.

JaMe76 · 2023-06-28T17:56:50Z

JaMe76
Jun 28, 2023
Maintainer Author

Script is out of date and will not work with latest release. Will be integrated into the built-in analyzer in next PR.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script for pipeline with structure recognition using table-transformer available here #116

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Script for pipeline with structure recognition using table-transformer available here #116

Uh oh!

JaMe76 Feb 9, 2023 Maintainer

Replies: 3 comments · 2 replies

Uh oh!

jayeshthukarul Feb 9, 2023

Uh oh!

buntys2010 Feb 28, 2023

`Downloading (…)lve/main/config.json: 100% 1.47k/1.47k [00:00<00:00, 71.9kB/s] You are using a model of type table-transformer to instantiate a model of type . This is not supported for all configurations of models and can yield errors.

Uh oh!

JaMe76 Feb 28, 2023 Maintainer Author

Uh oh!

buntys2010 Mar 1, 2023

Uh oh!

JaMe76 Jun 28, 2023 Maintainer Author

JaMe76
Feb 9, 2023
Maintainer

Replies: 3 comments 2 replies

jayeshthukarul
Feb 9, 2023

buntys2010
Feb 28, 2023

`Downloading (…)lve/main/config.json: 100%
1.47k/1.47k [00:00<00:00, 71.9kB/s]
You are using a model of type table-transformer to instantiate a model of type . This is not supported for all configurations of models and can yield errors.

JaMe76 Feb 28, 2023
Maintainer Author

JaMe76
Jun 28, 2023
Maintainer Author