Multimodal NLP-based email classification system for the construction industry.

Published at CIB W78 2024 (Marrakesh, Morocco): "Optimizing Email Classification in the Construction Industry through a Multimodal NLP Approach"

Digital Poststelle (Digital Mailroom) automates email classification for construction companies using a multimodal approach that processes both email text and attachments (PDFs, images, scanned documents, technical drawings).

The system combines BERT contextual embeddings with a BiLSTM layer and hierarchical attention mechanisms to classify emails into categories such as correspondence, contracts, invoices, legal documents, and more.

• BLIP for image/video captioning (converting visual attachments to text)
• LayoutLM for extracting text from scanned documents
• BERT + BiLSTM with sliding window for long email text processing
• Hierarchical attention with separate layers for major/minor classification tasks
• Outlook Add-in frontend (Vue.js 3) for end-user interaction

digital-poststelle/
├── Code/                # ML backend: classifier, API server, training
│   ├── email_classifier.py
│   ├── dp_api_new.py    # REST API server
│   ├── train_pl.py      # PyTorch Lightning training
│   ├── InfoExtractor.py # Email content extraction
│   └── config.yaml
├── Outlook Addin/       # Vue.js 3 frontend for Microsoft Outlook
│   ├── src/
│   ├── manifest.prod.xml
│   └── Dockerfile.prod
└── Dataset/             # Data processing and preparation tools
    ├── clean_msg.py
    └── getElement_multithread.py

Backend:

cd Code
conda env create -f mac_environment.yaml  # or ubuntu_environment.yaml
conda activate digital-poststelle
python dp_api_new.py --env dev

Frontend (Outlook Add-in):

cd "Outlook Addin"
npm install
npm run dev-server

Production deployment:

python dp_api_new.py --env prod
sudo sh deploy-prod.sh

• ML: PyTorch Lightning, Transformers (BERT, LayoutLM, BLIP), scikit-learn
• Backend: Python, REST API
• Frontend: Vue.js 3, Bootstrap 5, Webpack
• Deployment: Docker, Nginx

@inproceedings{zha2024optimizing,
  title     = {Optimizing Email Classification in the Construction Industry through a Multimodal NLP Approach},
  author    = {Zha, Yu and Ali, Sherief and Schumacher, Sebastian and Schulte, Michael and K{\"o}nig, Markus},
  booktitle = {Proceedings of the 41st International Conference of CIB W78},
  year      = {2024},
  address   = {Marrakesh, Morocco}
}

MIT License - Copyright (c) 2024 Yu Zha