We are indeed interested in creating smaller MoE models, but it is unlikely that we will train from scratch, as this is extremely computationally expensive.