I have tested SpherefaceNet-04 and SpherefaceNet-06 with and without BatchNorm.

Training on WebFace,
SpherefaceNet-04 WITHOUT BN can easily achieve ~98.00% (the paper reports 98.2%),
SpherefaceNet-04 WITH BN can easily achieve ~98.20% (fine-tuning 98.35%),
SpherefaceNet-06 WITHOUT BN can easily achieve ~98.50% (the paper doesnot give),
SpherefaceNet-06 WITH BN can easily achieve ~98.70% (fine-tuning 98.80%).

It should be noted that SpherefaceNet-04's forward only costs ~29ms (with BN ~33ms) and SpherefaceNet-06 costs ~49ms (with BN ~55ms), tested on E5-1650V4 3.6GHz, mini-caffe, openblas single thread.

The attachments are the nets with BN.

spherefacenet-04-06.zip