Improving BERT Model to Support for running with multi GPUs
You can simply use codes selection what would you use for running !

• Environment : Ubuntu 18.04, Tensorflow 1.14 CUDA 10.0, CUDNN 7.6.5
• Tested Model : BERT Multilingual Cased Model
• Dataset : SQuAD 1.1, 2.0 / KorQuAD 1.0, 2.1

• run_korquad : running script for KorQuAD 2.x datasets
  
• _tf_multi : use Tensorflow Mirror Strategy
  
• _hvd : use Horovod Framework
  

1. Tensorflow Mirrored Startegy API
   Use run_squad_tf_multi.py and optimization_tf_multi.py
   

2. Uber's Horovod Framework
   Use run_squad_hvd.py and optimization_tf_multi.py