o ߥi@sdZddlmZmZmZddlZddlmZddZdej d e fd d Z d eej j d eej j ej j ffd dZdeej j d eej j ej j ffddZdeed eefddZdS)z- Copyright 2020 The Microsoft DeepSpeed Team )DictListTupleN)MoEcCs@d}d}|D]\}}t|trd}|j}||fSq||fS)NFrT) named_modules isinstancer num_experts)mhas_moer _moduler[/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/gpt_moe/moe/utils.pyhas_moe_layers s rparamreturncCst|dr |js dSdS)N allreduceTF)hasattrr)rrrr is_moe_paramsrparamscCs:gg}}|D]}t|r||q||q||fS)N)rappend)r shared_params expert_paramsprrr*split_params_into_shared_and_expert_paramss   rgroupcCsVg}g}|D] }|jdur&t|r||j|jq||j|jq||fS)aSplit grad of parameters into grads of non-expert params and grads of expert params. This is useful while computing grad-norms for clipping and overflow detection group (List[torch.nn.Parameter]): Args: The group of parameters to split Returns: Tuple[List[torch.nn.Parameter], List[torch.nn.Parameter]]: list of gradients for non MoE params, list of gradients of MoE params N)gradrrtodtype)r expert_grads shared_gradsrrrr0split_params_grads_into_shared_and_expert_params)s r" param_groupsc Cst|tr t|}nt|tr|g}nt|ts!tdt|t}|D]}|dD] }t|r8||j q,q&t|}i}|D]O}i||d<|D]D}i||d|<|||d|d<d||d|d<| D]!}|dkr|dkrg||d||<qn||||d||<qnqLqB|D]'}g}|dD]}t|r||d|j d |q| |q||d<q| D]\}} | D] \} } | | qqt|S)zSplit parameters into different MoE groups for optimizer Args: param_groups (Tuple[Dict]): The list of parameter groups to split Returns: Tuple[Dict]: list of MoE/non-MoE groups for optimizer zUnknown param group type of rnameTmoe) rtuplelistdict ValueErrortypesetradd group_namekeysritems) r#data_parallel_group_names param_groupr group_moekeyori_key new_paramskvk1v1rrr4split_params_into_different_moe_groups_for_optimizerCsb              r:)__doc__typingrrrtorchlayerrrTensorboolrnn Parameterrr"r:rrrrs*