o wiV@sddlZddlmZmZmZddlmZddlZddl m Z ddl m Z ddl mZmZmZddlmZddlmZddlmZddlmZdd lmZd gZGd d d ejejejejZ dS) N)DictListOptional) model_summary) DictConfig OmegaConf open_dict)fn)io)$MaskedTokenLossReductionWithLossMask)logging)AppStateSpeechLanguageModelc seZdZfddZedefddZejdefddZd6d ede j fd d Z d e j fd dZd e j fddZd7deefddZddZddZddZddZeddZejddZed d!Zejd"d!Zdeeeeeejfffd#d$Zdeeeeeejfffd%d&Z 'd8d(eeeejfd)edeeeeeejfffd*d+Z 'd8d(eeeejfd)edeeeeeejfffd,d-Z d8d)edefd.d/Z!d8d)edefd0d1Z"ede#fd2d3Z$ede#fd4d5Z%Z&S)9rcsptt}|ti|_tj r$tj dur$tj |_ d|_ d|_ d|_d|_d|_d|_dSN)super__init__r _set_model_guidrcreate_cfgtorchcuda is_availablecurrent_device device_id_validation_step_outputs_validation_names_num_validation_dl_test_step_outputs _test_names _num_test_dl)self app_state __class__b/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/speechlm/models/base.pyr#s    zSpeechLanguageModel.__init__returncCs|jSrrr r$r$r%cfg8szSpeechLanguageModel.cfgr)cC ||_dSrr')r r)r$r$r%r)<  max_depthcCstj||dS)a Summarize this LightningModule. Args: max_depth: The maximum depth of layer nesting that the summary will include. A value of 0 turns the layer summary off. Default: 1. Return: The model summary object )r-)r summarize)r r-r$r$r%r.@szSpeechLanguageModel.summarizemodulecC|D]}d|_qdS)NF parameters requires_gradr r/paramr$r$r% freeze_moduleJ z!SpeechLanguageModel.freeze_modulecCr0)NTr1r4r$r$r%unfreeze_moduleNr7z#SpeechLanguageModel.unfreeze_moduleNstagecCsl|||t|j|jdur|jjj|j_Wdn1s(wYt | dS)zCalled at the beginning of fit, validate, test, or predict. This is called on every process when using DDP. Args: stage: fit, validate, test or predict N) propagate_model_guidsetup_multi_validation_datasetup_multi_test_datarrtrainer datamoduler)datar infor.)r r9r$r$r%setupRs  zSpeechLanguageModel.setupcCs2t|dst}tt|_||jdSdS)N model_guid)hasattrr struuiduuid4rBregister_model_guid)r appstater$r$r%res z#SpeechLanguageModel._set_model_guidcs<dtjffdd D] \}}j|_|qdS)zK Propagates the model GUID to all submodules, recursively. r/cs"j|_|D]}|qdSr)rBchildren)r/childrecursively_propagate_guidr r$r%rLss  zLSpeechLanguageModel.propagate_model_guid..recursively_propagate_guidN)plLightningModule named_modulesrB)r _r/r$rKr%r:ns  z(SpeechLanguageModel.propagate_model_guidcC|jdur tddS|jj}t|dddurtddSt|dd|_t|dd|_|jdur8t|j|_|jdura|jsKt dt|ddddt |jD|_td |jdSt|j|jkrxt d t|jd |jd dS) NzCSpeechLanguageModel.setup_multi_validation_data..zE`_validation_names` not found in data module. Setting default names: z%Number of validation names provided (z0) does not match number of validation datasets ().) r=r warningr>getattrr@rrlen ValueErrorranger data_moduler$r$r%r;|s2       z/SpeechLanguageModel.setup_multi_validation_datacCrQ) NrR_test_dsz9No test dataset found. Skipping setup of multi test data.rrz/`_num_test_dl` not found/valid in data module: cSrT)test_r$rVr$r$r%rYrZz=SpeechLanguageModel.setup_multi_test_data..z?`_test_names` not found in data module. Setting default names: zNumber of test names provided (z*) does not match number of test datasets (r[) r=r r\r>r]r@rrr^r_r`rar$r$r%r<s.      z)SpeechLanguageModel.setup_multi_test_datacCB|jdur|jSg|_|jdkrt|jD]}|jgq|jS)z Cached outputs of validation_step. It can be a list of items (for single data loader) or a list of lists (for multiple data loaders). Returns: List of outputs of validation_step. Nr,)rrr`appendr rPr$r$r%validation_step_outputss  z+SpeechLanguageModel.validation_step_outputscCr*r)rr valuer$r$r%rhr+cCre)z Cached outputs of test_step. It can be a list of items (for single data loader) or a list of lists (for multiple data loaders). Returns: List of outputs of test_step. Nr,)rrr`rfrgr$r$r%test_step_outputss  z%SpeechLanguageModel.test_step_outputscCr*r)rrir$r$r%rkr+cCs|jdurt|jdkriSt|jdtr7|j|jdd}|dur0d|vr0|j|ddd|j|Sdii}t|jD]\}}| |}|j||d}|pSi}d|vrgd|vrg||j krg|d|d<| D]N\}}|dkri}| D]&\} } | |dvr||j kr| } | ||d| <n|d| } | || <qy|d} | || |d<qk|d|} ||| <qk|j|q@d|vr|j|ddd|S) a Default DataLoader for Validation set which automatically supports multiple data loaders via `multi_validation_epoch_end`. If multi dataset support is not required, override this method entirely in base class. In such a case, there is no need to implement `multi_validation_epoch_end` either. .. note:: If more than one data loader exists, and they all provide `val_loss`, only the `val_loss` of the first data loader will be used by default. This default can be changed by passing the special key `val_dl_idx: int` inside the `validation_ds` config. Args: outputs: Single or nested list of tensor outputs from one or more data loaders. Returns: A dictionary containing the union of all items from individual data_loaders, along with merged logs from all data loaders. Nrdataloader_idxlogTon_epochval_lossrP) rhr^ isinstancedictmulti_validation_epoch_endlog_dictpopclear enumerate get_validation_dataloader_prefix _val_dl_idxitemsupdate)r output_dictrm val_outputsdataloader_prefixdataloader_logskvruk_logv_log new_k_log output_logsnew_kr$r$r%on_validation_epoch_endsD       z+SpeechLanguageModel.on_validation_epoch_endcCs|jdurt|jdkriSt|jdtr7|j|jdd}|dur0d|vr0|j|ddd|j|Sdii}t|jD]\}}| |}|j||d}|pSi}d|vrgd|vrg||j krg|d|d<| D]P\}}|dkri}| D]&\} } | |dvr||j kr| } | ||d| <n|d| } | || <qy| di} | || |d<qk|d|} ||| <qk|j|q@d|vr|j|ddd|S) a Default DataLoader for Test set which automatically supports multiple data loaders via `multi_test_epoch_end`. If multi dataset support is not required, override this method entirely in base class. In such a case, there is no need to implement `multi_test_epoch_end` either. .. note:: If more than one data loader exists, and they all provide `test_loss`, only the `test_loss` of the first data loader will be used by default. This default can be changed by passing the special key `test_dl_idx: int` inside the `test_ds` config. Args: outputs: Single or nested list of tensor outputs from one or more data loaders. Returns: A dictionary containing the union of all items from individual data_loaders, along with merged logs from all data loaders. NrrlrnTro test_lossrP)rkr^rrrsmulti_test_epoch_endrurvrwrxget_test_dataloader_prefix _test_dl_idxr{getr|)r r}rm test_outputsrrrrrurrrrrr$r$r%on_test_epoch_endJsD        z%SpeechLanguageModel.on_test_epoch_endroutputsrmcCtddS)a8 Adds support for multiple validation datasets. Should be overriden by subclass, so as to obtain appropriate logs for each of the dataloaders. Args: outputs: Same as that provided by LightningModule.on_validation_epoch_end() for a single dataloader. dataloader_idx: int representing the index of the dataloader. Returns: A dictionary of values, optionally containing a sub-dict `log`, such that the values in the log will be pre-pended by the dataloader prefix. aEMulti data loader support has been enabled, but `multi_validation_epoch_end(outputs, dataloader_idx) has not been implemented. If you require multi data loader support for validation sets, please override this method. If you do not require multi data loader support, please instead override `on_validation_epoch_end(outputs).Nr r\r rrmr$r$r%rtz.SpeechLanguageModel.multi_validation_epoch_endcCr)a2 Adds support for multiple test datasets. Should be overriden by subclass, so as to obtain appropriate logs for each of the dataloaders. Args: outputs: Same as that provided by LightningModule.on_validation_epoch_end() for a single dataloader. dataloader_idx: int representing the index of the dataloader. Returns: A dictionary of values, optionally containing a sub-dict `log`, such that the values in the log will be pre-pended by the dataloader prefix. a9Multi data loader support has been enabled, but `multi_test_epoch_end(outputs, dataloader_idx) has not been implemented. If you require multi data loader support for validation sets, please override this method. If you do not require multi data loader support, please instead override `on_test_epoch_end(outputs).Nrrr$r$r%rrz(SpeechLanguageModel.multi_test_epoch_endcC |j|Sz Get the name of one or more data loaders, which will be prepended to all logs. Args: dataloader_idx: Index of the data loader. Returns: str name of the data loader at index provided. )rr rmr$r$r%ry z4SpeechLanguageModel.get_validation_dataloader_prefixcCrr)rrr$r$r%rrz.SpeechLanguageModel.get_test_dataloader_prefixcCs|jst|_|jSr)_training_loss_reductionr r(r$r$r%training_loss_reductionsz+SpeechLanguageModel.training_loss_reductioncCs|js tdd|_|jS)NT)validation_step)_validation_loss_reductionr r(r$r$r%validation_loss_reductions z-SpeechLanguageModel.validation_loss_reduction)r,r)r)'__name__ __module__ __qualname__rpropertyrr)setterintr ModelSummaryr.nnModuler6r8rrDrArr:r;r<rhrkrrTensorrrrrtrryrr rr __classcell__r$r$r"r%r"s\       $$_^    )!rEtypingrrrlightning.pytorchpytorchrMrtorch.nnrlightning.pytorch.utilitiesr omegaconfrrrnemo.collections.llmr nemo.lightningr nemo.lightning.megatron_parallelr nemo.utilsr nemo.utils.app_stater __all__rNIOMixinConnectorMixinFNMixinrr$r$r$r%s        "