o Ti@sddlZddZddZdS)Nc Csg}g}t|dD],}tj|d|jdd|}|d}||||||d|ddfq tj|dd}|ddddd|d|f}|||fS)Nr)device)dim)rangesizetorchrandpermrsortappendcat) reserved_length hidden_statesmask rand_listpart_hidden_stateskB_tmpB part_maskrf/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/data_pipeline/data_routing/utils.pybsh_decoder_gather s  "  rcCs@t|dD]}||ddddf||||ddf<q|S)Nr)rr)r rrrrrrbsh_decoder_scatters*r)rrrrrrrs