o }oi @sddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZed \ZZhd ZeGdddZ  dde jdedejdBde jdBde jf ddZde jde ddfddZ!dS)) dataclass)partialN)dist_checkpointing) lightning)llm)ckpt_to_context_subdir)TrainerContextckpt_to_weights_subdir)logging)is_global_rank_zero) safe_importzmodelopt.torch.prune> num_layers hidden_sizeffn_hidden_sizenum_query_groupsnum_attention_headsc@s~eZdZUdZdZedBed<dZedBed<dZedBed<dZ edBed<dZ edBed<dZ e edBed<d d Z dS) PruningConfiga\Pruning parameters. None means no pruning of the corresponding dimension. Args: target_ffn_hidden_size (int, optional): Target size of MLP FFN hidden dimension. target_hidden_size (int, optional): Target size of embedding hidden dimension. target_num_attention_heads (int, optional): Target number of attention heads. Required if `target_num_query_groups` is provided. target_num_query_groups (int, optional): Target number of query groups for grouped-query attention. Required if `target_num_attention_heads` is provided. target_num_layers (int, optional): Target number of transformer layers using importance metric. drop_layers (list[int], optional): List of specific layer indices (1-indexed) to drop from the model. Cannot be used with other pruning parameters. Ntarget_ffn_hidden_sizetarget_hidden_sizetarget_num_attention_headstarget_num_query_groupstarget_num_layers drop_layerscCs@|jr|j|j|j|j|jg}tdd|DrtddSdS)Ncss|]}|duVqdS)N).0prr^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/modelopt/prune/pruner.py Jsz.PruningConfig.__post_init__..z8drop_layers cannot be used with other pruning parameters)rrrrrrany ValueError)self other_paramsrrr __post_init__AszPruningConfig.__post_init__)__name__ __module__ __qualname____doc__rint__annotations__rrrrrlistr"rrrrr*s  rmodelpruning_config data_moduletrainerreturnc stsJdjrtjjj|jd|S|dusJd|dus%Jd|j|_t dfddt D}tj |d d |idd t t j||d d id|S)aPrune a GPT model in-place based on the provided pruning configuration. Args: model (llm.GPTModel): The model to prune. pruning_config (PruningConfig): The pruning configuration. data_module (pl.LightningDataModule, optional): The data module to use for pruning. Required if not dropping layers. trainer (nl.Trainer, optional): The trainer to use for pruning. Required if not dropping layers. Returns: llm.GPTModel: The pruned model. z/nvidia-modelopt is required to prune the model.)layers_to_dropNz+data_module is required to prune the model.z'trainer is required to prune the model.zPruning model...cs2i|]}td|dur|td|qS)target_N)getattr)rkr+rr ks z#prune_gpt_model..mcore_gpt_minitron export_config forward_loopr*)datar- tokenizer)mode constraints dummy_inputconfig) HAVE_MODELOPTrmtppluginsmegatrondrop_mcore_gpt_layerstrain_dataloaderval_dataloaderr infoSUPPORTED_PRUNING_HPARAMSprunerrvalidate)r*r+r,r-r6rr3rprune_gpt_modelNs&   rI save_pathcCstd|dtdhBD]}t|jjj|t|jj|qt|dd}|j dddt |j j |trFt|jt|dgdtd |d d S) z"Save pruned model nemo checkpoint.zSaving pruned model to z... kv_channelsT) is_saving)parentsexist_okr*) yaml_attrszPruned model saved to  N)r rErFsetattrr*__io__r=r1r mkdirrsavestrategymegatron_parallelsharded_state_dictr r from_trainerio_dumpr)r-rJr2 weight_pathrrrsave_pruned_model{s r[)NN)" dataclassesr functoolsrpytorch_lightningpl megatron.corernemornlnemo.collectionsrnemo.lightning.ckpt_utilsrnemo.lightning.io.plrr nemo.utilsr nemo.utils.get_rankr nemo.utils.import_utilsr r?r>rFrGPTModelLightningDataModuleTrainerrIstrr[rrrrs:           & -