I have a question if MultiBoxLoss function reflects false negatives (and possibly false positives)?

I'm trying to optimize my SSD detection model and while my optimizations provide lower training loss and higher testing accuracy, the result have more missing detections than before.

The multibox loss used in the SSD method is defined as L = Lconf + alpha * Lloc, but what happens when the labelled object is not detected at all? How is it reflected in the computed loss? Does it increase the loss or is it completely ignored?

I'm thinking to update the loss function to be something like L = Lconf + alpha * Lloc + beta * abs(number_of_detections - number_of_gt), but I don't know if it makes any sense.

Thank you.