o }oi'@sjddlZddlmZmZmZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZdd lmZmZgdZddZd(de dee!fddZ"     d)dee#dee!dee!deede de f ddZ$        d*de!d ee!d!e dee#d"ee!d#e d$e d%e%deeeffd&d'Z&dS)+N)ListOptionalUnion)asdict)Trainer) DictConfig) BertModule) DecoderModule) EncoderModule)get_huggingface_lm_model)get_huggingface_pretrained_lm_models_list)#get_megatron_pretrained_bert_models)NeMoTransformerConfig)get_huggingface_transformerget_nemo_transformer)AppStatelogging)get_pretrained_lm_models_list get_lm_model pad_batchcCs^g}tdd|D}|D]}t|}|||kr%||g|||||q ||fS)NcSsg|]}t|qS)len).0tokensrr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/lm_utils.py +szpad_batch..)maxrextendappend)batchpad_idmax_lencontext_lengthsmax_context_lengthrcontext_lengthrrrr)s  rFinclude_externalreturncCs t|dS)z Returns the list of supported pretrained model names Args: include_external if true includes all HuggingFace model names, not only those supported language models in NeMo. r%)r r'rrrr4s r config_dict config_file vocab_filetrainercfgc s|jdr|jjr|jjtddvrt|jjd|r*|r*td|dd}|dr?|jddr?|jdd}d d lmd tt ffd d }|}|j durj|j dddurjd|j ddvsn||vrd dl }d d lmGfddd|j j |jdrj|jj|d} n j|jd|d} | j_| j_| jj_| St|||jjd} |jdrt} | jstj|jjst|jjd| j|jjd| S)a~ Helper function to instantiate a language model encoder, either from scratch or a pretrained model. If only pretrained_model_name are passed, a pretrained model is returned. If a configuration is passed, whether as a file or dictionary, the model is initialized with random weights. Args: config_dict: path to the model configuration dictionary config_file: path to the model configuration file vocab_file: path to vocab_file to be used with Megatron-LM trainer: an instance of a PyTorch Lightning trainer cfg: a model configuration Returns: Pretrained BertModule pretrained_model_nameFr'zk is not in get_pretrained_lm_models_list(include_external=False), will be using AutoModel from HuggingFace.zLBoth config_dict and config_file were found, defaulting to use config_file: z will be used.language_modelrMegatronBertModelr&csddD}|S)NcSsg|]}|jqSr)r-)rmodelrrrrnszMget_lm_model..get_megatron_pretrained_bert_models..)list_available_models)#all_pretrained_megatron_bert_modelsr0rrr lsz9get_lm_model..get_megatron_pretrained_bert_modelsNtokenizer_namemegatroncs&eZdZfddZddZZS)zget_lm_model..Identitycst|dSN)super__init__)self)Identity __class__rrr9~sz'get_lm_model..Identity.__init__cWs|Sr7r)r:xargsrrrforwardsz&get_lm_model..Identity.forward)__name__ __module__ __qualname__r9r? __classcell__r)r;)r<rr;}sr; lm_checkpoint) restore_pathr+)r+)r(r)r-z not found)rE)r/getr-rrwarningAnemo.collections.nlp.models.language_modeling.megatron_bert_modelr1rstr tokenizertorchnnModule restore_fromrDfrom_pretrainedr2lm_head binary_headpoolerr ris_model_being_restoredospathexists ValueErrorrestore_weights) r(r)r*r+r,pretrain_model_namer r4rKr2 app_stater)r;r1rr?sT           rnemoTlibrary model_name pretrainedcheckpoint_fileencoderpre_ln_final_layer_norm padding_idxc Csd}|dkr)t|trt|}t||||||d}|dur'tj|r'td|S|dkr7t||||d}|S|dkr?tdtd ) aGets Transformer based model to be used as an Encoder or Decoder in NeMo NLP. First choose the library to get the transformer from. This can be huggingface, megatron, or nemo. Use the model_name arg to get a named model architecture and use the pretrained arg to get the named model architecture with pretrained weights. If model_name is None, then we can pass in a custom configuration via the config_dict. For example, to instantiate a HuggingFace BERT model with custom configuration we would do: encoder = get_transformer(library='huggingface', config_dict={ '_target_': 'transformers.BertConfig', 'hidden_size': 1536 }) Args: library (str, optional): Can be 'nemo', 'huggingface', or 'megatron'. Defaults to 'nemo'. model_name (Optional[str], optional): Named model architecture from the chosen library. Defaults to None. pretrained (bool, optional): Use True to get pretrained weights. False will use the same architecture but with randomly initialized weights. Defaults to False. config_dict (Optional[dict], optional): Use for custom configuration of transformer. Defaults to None. checkpoint_file (Optional[str], optional): Provide weights for the transformer from a local checkpoint. Defaults to None. encoder (bool, optional): True returns an EncoderModule, False returns a DecoderModule. Defaults to True. Returns: Union[EncoderModule, DecoderModule]: Ensures that Encoder/Decoder will work in EncDecNLPModel Nr[)r]r^r(r`rarbzNLoading transformer weights from checkpoint file has not been implemented yet. huggingface)r]r^r(r`r6z[megatron-lm bert support has been deprecated in NeMo 1.5+. Please use NeMo 1.4 for support.z2Libary must be 'nemo', 'huggingface' or 'megatron') isinstancerrrrTrUisfilerWr) r\r]r^r(r_r`rarbr2rrrget_transformers4&   rf)F)NNNNN)r[NFNNTTr)'rTtypingrrrattrrlightning.pytorchr omegaconfr/nemo.collections.nlp.modules.common.bert_moduler2nemo.collections.nlp.modules.common.decoder_moduler 2nemo.collections.nlp.modules.common.encoder_moduler Anemo.collections.nlp.modules.common.huggingface.huggingface_utilsr r ;nemo.collections.nlp.modules.common.megatron.megatron_utilsr ;nemo.collections.nlp.modules.common.transformer.transformerrAnemo.collections.nlp.modules.common.transformer.transformer_utilsrr nemo.utilsrr__all__rboolrIrdictrintrfrrrrsz           i