o wi @sddlmZddlmZddlmmZddlZ ddl m Z ddl m ZddlmZddlmZddlmZmZdd lmZdd lmZhd ZeGd d d Z  ddejdede jdBdejdBdejf ddZdejde ddfddZ!dS)) dataclass)partialN)dist_checkpointing) lightning)llm)ckpt_to_context_subdir)TrainerContextckpt_to_weights_subdir)logging)is_global_rank_zero> num_layers hidden_sizemamba_head_dimffn_hidden_sizemamba_num_headsnum_query_groupsnum_attention_headsc@seZdZUdZdZedBed<dZedBed<dZedBed<dZ edBed<dZ edBed<dZ edBed<dZ edBed <dZ eedBed <d d ZdS) PruningConfigaPruning parameters. None means no pruning of the corresponding dimension. Args: target_ffn_hidden_size (int, optional): Target size of MLP FFN hidden dimension. target_hidden_size (int, optional): Target size of embedding hidden dimension. target_num_attention_heads (int, optional): Target number of attention heads. Required if `target_num_query_groups` is provided. target_num_query_groups (int, optional): Target number of query groups for grouped-query attention. Required if `target_num_attention_heads` is provided. target_mamba_num_heads (int, optional): Target number of Mamba attention heads. target_mamba_head_dim (int, optional): Target dimension of Mamba attention heads. target_num_layers (int, optional): Target number of transformer layers using importance metric. drop_layers (list[int], optional): List of specific layer indices (1-indexed) to drop from the model. Cannot be used with other pruning parameters. Ntarget_ffn_hidden_sizetarget_hidden_sizetarget_num_attention_headstarget_num_query_groupstarget_mamba_num_headstarget_mamba_head_dimtarget_num_layers drop_layerscCsH|jr |j|j|j|j|j|j|jg}tdd|Dr"t ddSdS)Ncss|]}|duVqdS)N).0prrg/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/modelopt/prune/pruner.py Psz.PruningConfig.__post_init__..z8drop_layers cannot be used with other pruning parameters) rrrrrrrrany ValueError)self other_paramsrrr __post_init__Es  zPruningConfig.__post_init__)__name__ __module__ __qualname____doc__rint__annotations__rrrrrrrlistr%rrrrr*s  rmodelpruning_config data_moduletrainerreturnc sjrtjj|jd|S|dusJd|dusJd|j|_tdfddtD}tj |dd |idd t t j ||d d id |S)aPrune a GPT / Mamba (sub-class of GPT) model in-place based on the provided pruning configuration. Args: model (llm.GPTModel): The model to prune. pruning_config (PruningConfig): The pruning configuration. data_module (pl.LightningDataModule, optional): The data module to use for pruning. Required if not dropping layers. trainer (nl.Trainer, optional): The trainer to use for pruning. Required if not dropping layers. Returns: llm.GPTModel: The pruned model. )layers_to_dropNz+data_module is required to prune the model.z'trainer is required to prune the model.zPruning model...cs2i|]}td|dur|td|qS)target_N)getattr)rkr.rr ps z(prune_language_model..mcore_minitron export_config forward_loopr-)datar0 tokenizer)mode constraints dummy_inputconfig) rmtpplugins drop_mcore_language_model_layerstrain_dataloaderval_dataloaderr infoSUPPORTED_PRUNING_HPARAMSprunerrvalidate)r-r.r/r0r9rr6rprune_language_modelTs$  rJ save_pathcCstd|dtdhBD]}t|jjj|t|jj|qt|dd}|j dddt j |j j ||j jdtrJt|jt|dgd td |d d S) z"Save pruned model nemo checkpoint.zSaving pruned model to z... kv_channelsT) is_saving)parentsexist_ok)content_metadatar-) yaml_attrszPruned model saved to  N)r rFrGsetattrr-__io__r@r4r mkdirrsavestrategymegatron_parallelsharded_state_dictsharded_state_dict_metadatar r from_trainerio_dumpr)r0rKr5 weight_pathrrrsave_pruned_models  r^)NN)" dataclassesr functoolsrmodelopt.torch.prunetorchrHrApytorch_lightningpl megatron.corernemornlnemo.collectionsrnemo.lightning.ckpt_utilsrnemo.lightning.io.plrr nemo.utilsr nemo.utils.get_rankr rGrGPTModelLightningDataModuleTrainerrJstrr^rrrrs8         , ,