@codebecker If you have solved this problem, can you tell me how you solved it?

TLDR: The error appear because The number of microbatches should evenly divide the batch size which did not work out in my case so I set the microbatches parameter to 1 to avoid this error to the cost of performance.

I used a batch size of 250 which yield the error of Dimension size must be evenly divisible by 250 because the batch size was not dividable by 250. However I changed the number of training images to 5000 and the batch size to 500 which should have clearly fixed this issue but the error still continues to appear. I am not sure if the error source is in my code or the implementation (as my changes should have fixed it) however I found a "workaround" which is to set the parameter of microbatches to 1. As described in the cited explanation of the microbatches from the offical implementation example this is the default setting but comes with a higher cost of performance.

microbatches (int) - Each batch of data is split in smaller units called microbatches. By default, each microbatch should contain a single training example. This allows us to clip gradients on a per-example basis rather than after they have been averaged across the minibatch. This in turn decreases the (negative) effect of clipping on signal found in the gradient and typically maximizes utility. However, computational overhead can be reduced by increasing the size of microbatches to include more than one training examples. The average gradient across these multiple training examples is then clipped. The total number of examples consumed in a batch, i.e., one step of gradient descent, remains the same. The number of microbatches should evenly divide the batch size.

@NOS9512: I hope that helps wish you a merry christmas 🎄

I faced a similar problem and training is very slow with the microbtach set to 1

Any update to make training possible with a bigger size of the microbatch?