o }oib@sddlZddlmZddlmZddlmZmZddlZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZdd lmZmZmZmZmZmZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0Gddde e%Z1 d dejdejde2de2de3ejdeejde4ejeejejffddZ5dS)!N) defaultdict)repeat)AnyOptional)collate_vectors)LightningModule) DictConfig) PeftModel)Tensor) fully_shard) ReplicateShard)ColwiseParallelPrepareModuleInputRowwiseParallelSequenceParallel loss_parallelparallelize_module)GenerationConfig)PromptFormatter) AutoTokenizer) HFHubMixin)maybe_install_lora)configure_optimizers is_frozen)load_pretrained_hfmove_embeddingsetup_speech_encoder)loggingc s`eZdZd3fdd ZeddZedefddZedefd d Zedefd d Z edefd dZ  d4de de de e e ffddZde fddZde defddZd3ddZd3ddZde defddZd3dd Zd3d!d"Zd#ed$efd%d&Zfd'd(Ze   d5d)eee e d*ej d+ej d,edej f d-d.Zd/d0Zd3d1d2ZZ S)6SALMreturnNcst|tsJdt|dt|t||_|jj|_t |jj dd|_ |j d|jgit |jj |jjd|_|jjj|_|jj`t|t|d|_d|_dS)NzYou must pass the config to SALM as a Python dict to support hyperparameter serialization in PTL checkpoints (we got: 'type(cfg)=z').T)use_fastadditional_special_tokens)pretrained_weightsF) isinstancedicttypesuper__init__save_hyperparametersrcfgaudio_locator_tagrpretrained_llm tokenizeradd_special_tokensrr#llmmodel embed_tokensrr _use_fsdp_use_tp)selfr* __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/speechlm2/models/salm.pyr(/s$      z SALM.__init__cC|jjS)z&Return the size of the text tokenizer.)r1num_embeddingsr4r7r7r8text_vocab_sizeHszSALM.text_vocab_sizecCr9N)r-bos_idr;r7r7r8 text_bos_idMzSALM.text_bos_idcCr9r=)r-eos_idr;r7r7r8 text_eos_idQr@zSALM.text_eos_idcCs2|jj}|dur |jj}|durtdd}|S)Nzqthe text tokenizer has no or tokens available, using id 0 for padding (this may lead to silent bugs).r)r-padunk_idwarningswarn)r4pad_idr7r7r8 text_pad_idUszSALM.text_pad_idcCs|j|jSr=)r- token_to_idr+r;r7r7r8audio_locator_tag_idaszSALM.audio_locator_tag_id input_embedsattention_maskcCs<|j||||dudd}d|di}|dur|d|d<|S)z Implements a fully offline forward pass through the entire model. The flow is the following: |speech and text embeddings| -> |llm| -> |lm_head| -> |token ids| NT) inputs_embedsrLpast_key_values use_cache return_dictlogitsrNcache)r/)r4rKrLrRoutansr7r7r8forwardes  z SALM.forwardbatchc CsB|j|d|dd\}}ddt||D}t|d|jkd|d}||}t|d||j|j||d|dd d \}}}|d d d d f}|d d d d f}|d d d d f}|jr|j d } |j d d | } dkr|d d d | f}|d d d | f}|d d d | f}|||||dS)a Performs additional processing on the mini-batch collected from dataloader. Notably: * Convert source audio to speech representations. * Convert target audio to target audio tokens. * Convert target text to embeddings. * Combine the input audio and target text embeddings. * Take care of any necessary slicing to align the shapes of source audio, target audio, and target token ids. audios audio_lens) input_signalinput_signal_lengthcSsg|] \}}|d|qSr=r7).0embemblenr7r7r8 z'SALM.prepare_inputs.. input_idsr loss_maskr`embeds padding_idplaceholder_id replacements target_idsNtensor_parallel) audio_embeds text_embedsrKrLrh) perceptionziptorchwhererJr1&replace_placeholders_and_build_targetsrHr3 device_meshsizeshape) r4rV audio_embsaudio_emb_lensinput_ids_to_embed text_embs input_embsrhrL tp_world_size remainderr7r7r8prepare_inputss:    zSALM.prepare_inputs batch_idxc CsD|jj|jj|jfD] }t|r|q ||}||d|dd}|ddk}t !t j j j |ddd|dddd dd |}Wdn1sUwY|djdd \}} |t |jdurx|jjdjdd nd|| |t j||| |d |jk|d d} |j| dd| S)NrKrLrLrhrbrQrrjsum reduction ignore_indexlrr`)loss learning_rate batch_sizesequence_length num_framestarget_to_input_ratio padding_ratioT)on_step)rn preprocessorencoderr/revalr}longrrrpnn functional cross_entropyflattenru as_tensor_trainertrainer optimizers param_groupstofloat32rHnumellog_dict) r4rVr~minputsforward_outputsrrBTrTr7r7r8 training_steps:  (    zSALM.training_stepcCstt|_tt|_dSr=)rlist_partial_val_losses_partial_accuraciesr;r7r7r8on_validation_epoch_starts zSALM.on_validation_epoch_startcCsg}|jD]\}}t|}|jd||ddd||q|jdt|dddg}|jD]\}}t|}|jd||ddd||q9|jdt|ddd|j|jdS)N val_loss_T)on_epoch sync_distval_lossval_acc_val_acc) ritemsrpstackmeanlogappendrclear)r4 val_lossesnamevalsr accuraciesaccsrr7r7r8on_validation_epoch_ends   zSALM.on_validation_epoch_endc Cs|D]\}}|dur q||}||d|dd}|ddk}t!tjjj|d dd|d ddd dd |}Wdn1sMwY|dj d d  d } |d d } | | dk} | | dk} | |  } |j|| |j||qdS) NrKrLrrhrbrQrrjrrridim)rr}rrrrprrrrargmaxvieweqfloatrrrr) r4rVr~r dataset_batchrrrrpredsrefsaccuracyr7r7r8validation_steps2    zSALM.validation_stepcC|Sr=)rr;r7r7r8on_test_epoch_startzSALM.on_test_epoch_startcCrr=)rr;r7r7r8on_test_epoch_end rzSALM.on_test_epoch_endargskwargscOs|j|i|Sr=)rr4rrr7r7r8 test_step szSALM.test_stepcs>ttj|i|WddS1swYdSr=)rr'backwardrr5r7r8rs"z SALM.backwardpromptsrWrXgeneration_configcstjjjtfdd|Djdj}|durO| |j kd} |}fddt ||D}t ||jj |dd\} } } | | d} n |jk} || d } tjjd i| d |i} Wd| S1svwY| S) a Generate LLM answers given text or mixed text+audio prompts. Inputs: prompts: batch of prompts as list[dict] each in the following format [ # batch example id 0 [{"role": "user"}, "slots": {"message": "Repeat after me, translating to Polish."}] # batch example id 1 [{"role": "user"}, "slots": {"message": "Repeat after me."}] ] "role" is LLM-specific, you can pass multiple turns as well. audios: Optional. Time-domain audio signal zero-padded batch of shape (B, T). The number of audios must correspond to the number of occurrences of in prompts. Each prompt can have multiple audios. audio_lens: Optional. Length of each audio example. generation_config: Optional HuggingFace GenerationConfig object. csg|] }j|ddqS))turnsr`) encode_dialog)r[prompt) formatterr7r8r^1r_z!SALM.generate..) padding_valueNrcs0g|]\}}|d|dddqS)r)rn unsqueeze)r[aalr;r7r8r^<s$rc)rMrL)r`rLrr7)rresolver* prompt_formatr-rrHrdevicerqrJr1rorrrr/generate)r4rrWrXrtokenstokens_to_embed token_embedsrlrK_rLgeneration_inputs answer_tokensr7)rr4r8rsB        z SALM.generatecCst|Sr=)rr;r7r7r8rVrzSALM.configure_optimizersc Cs|j}|dur dS|j}t|tr|jj}|d}dkrd|_tt ft dfddt d}t ||||jj D]_}t ttttt ddt tt dft fdtttt ddd }|j}d D]*}t||}||d krtd |d |d |dt||||qjt |||q(sz9replace_placeholders_and_build_targets..)rr)rtrrrangenonzerolenrclonerpfullronesboolcat ValueErrormaxzerosrrro) r`rdrerfrgrhrseq_len hidden_dimrrroutput_sequencesoutput_target_idsoutput_att_masksreplacement_idxiplaceholder_positionsnew_target_idssegmentstarget_segments att_masksprev_pospossegment_target_idsrepmax_seq_lengthoutputattention_masksrtgtattr7r7r8rrs        "            rrr=)6rE collectionsr itertoolsrtypingrrrplhotse.dataset.collationr lightningr omegaconfrpeftr r torch.distributed.fsdpr torch.distributed.tensorr r !torch.distributed.tensor.parallelrrrrrr transformersrnemo.collections.common.promptsr"nemo.collections.common.tokenizersr'nemo.collections.speechlm2.parts.hf_hubr%nemo.collections.speechlm2.parts.lorar,nemo.collections.speechlm2.parts.optim_setuprr+nemo.collections.speechlm2.parts.pretrainedrrr nemo.utilsrrr rtuplerrr7r7r7r8sR