get mask token from tokenizer (deepspeedai#2592)

Fix invalid check of recorded parameter orders in zero stage3. (deeps…

…peedai#2550)

Co-authored-by: Olatunji Ruwase <[email protected]>

Fix typos: deepseed -> deepspeed (deepspeedai#2499)

Fix broken link to DeepSpeed Megatron fork (deepspeedai#2440)

Co-authored-by: Lev Kurilenko <[email protected]>

Add more options to inference benchmark (deepspeedai#2325)

fix deepspeedai#2240: wrong time unit in flops_profiler (deepspeedai#…

…2241)

Co-authored-by: Olatunji Ruwase <[email protected]>
Co-authored-by: Jeff Rasley <[email protected]>

fix ds-inference without policy (deepspeedai#2247)

Co-authored-by: Jeff Rasley <[email protected]>

Tensor parallelism for Mixture of Experts (deepspeedai#2074)

* tensor parallelism for mixture of experts
Co-authored-by: Olatunji Ruwase <[email protected]>
Co-authored-by: Ammar Ahmad Awan <[email protected]>

use HF NeoX (deepspeedai#2087)

Co-authored-by: Olatunji Ruwase <[email protected]>
Co-authored-by: Jeff Rasley <[email protected]>

Improving memory utilization of Z2+MoE (deepspeedai#2079)

* Shards expert parameter groups
* Do upscaling, optimizer and deletion of fp32 grads one-by-one on each parameter group in zero-2
Co-authored-by: Olatunji Ruwase <[email protected]>