o Ti+ @sVddlmZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl m Z de jdeeeffd d Zd e jdefd d Zdee j jdeee j jee j jffddZdee j jdeee jee jffddZ ddeeeefeeeefdfeeeeffdeeefdeeeeffddZddZdefddZdS) ) defaultdict)AnyDictListSetTupleUnioncastN)nn)MoEmreturncCs<d}d}|D]}t|trd}|j}||fSq||fS)NFrT)modules isinstancer num_experts)r has_moermodulerG/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/moe/utils.pyhas_moe_layerss  rparamcCst|dr |js dSdS)N allreduceTF)hasattrr)rrrr is_moe_paramsrparamscCs8g}g}|D]}t|r||q||q||fSN)rappend)r shared_params expert_paramsprrr*split_params_into_shared_and_expert_params!s  r!groupcCsVg}g}|D] }|jdur&t|r||j|jq||j|jq||fS)aSplit grad of parameters into grads of non-expert params and grads of expert params. This is useful while computing grad-norms for clipping and overflow detection group (List[torch.nn.Parameter]): Args: The group of parameters to split Returns: Tuple[List[torch.Tensor], List[torch.Tensor]]: list of gradients for non MoE params, list of gradients of MoE params N)gradrrtodtype)r" expert_grads shared_gradsr rrr0split_params_grads_into_shared_and_expert_params.s r(髪 param_groups.max_group_sizec Cst|tr t|}nt|tr|g}nt|ts!tdt|t}|D]}ttt j |dD] }t |r>| |j q2q&tdd}|D]}|D]}i||dgd||d|<qLqH|D]-}g}ttt j |dD]}t |r||d|j d|qp||qp||d<qb|dur|D]S}|D]L}g} g} d } ttt j |dD]"}| ||kr| || |7} q| | |g} |} q| r| | | D] } |i|d| iqqq|S|D]}|D]}||qq|S) a6Split parameters into different MoE groups for optimizer Args: param_groups (Union[Dict[str, Any], Tuple[Dict[str, Any], ...], List[Dict[str, Any]]]) The list of parameter groups to split Returns: List[Dict[str, Any]]: list of MoE/non-MoE groups for optimizer zUnknown param group type of rcSsttSr)rdictrrrrdszFsplit_params_into_different_moe_groups_for_optimizer..T)namemoerr.Nr)rtuplelistr, ValueErrortypesetr rr Parameterradd group_namerrvaluesnumel) r*r+data_parallel_group_names param_groupr group_moekey new_params moe_group cur_group all_groupssize_of_cur_groupr"rrr4split_params_into_different_moe_groups_for_optimizerHsp                rCcCs |ddS)Nr/F)get)r;rrris_moe_param_groups rEmodel_parameterscCst|ts Jd|D]}t|tjtfstdt|q |d}t|tjjr3|dd}t |St|trGt dd|DsEt |S|SdS)Nzmodel_parameters must be a listzgparam argument that would be given to the optimizer should be an iterable of Tensors or dicts, but got rz dense-params)rr.cSsg|]}d|vqS)r/r).0r;rrr sz.configure_moe_param_groups..) rr1torchTensorr, TypeErrorr3r r5rCany)rFr firstr;rrrconfigure_moe_param_groupss   rN)r)) collectionsrtypingrrrrrrr rIr layerr ModuleboolintrrJrr5r!r(strfloatrCrErNrrrrs6 $       0  O