o wi|@s(ddlZddlmZddlmZddlmZddlmZm Z ddl Z ddl m Z ddl mZddlmZdd lmZdd l mZdd lmZdd lmZmZdd lmZmZmZmZmZmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:Gdddee)Z; d*de jde jdee je e je jffd!d"Z?de jde jde e jdee je je e jff d#d$Z@d%e=e=eAd&ee je jfdBfd(d)ZDdS)+N) defaultdict)repeat)Path)AnyOptional)CutSet)LightningModule) DictConfig) PeftModel)Tensor) fully_shard) ReplicateShard)ColwiseParallelPrepareModuleInputRowwiseParallelSequenceParallel loss_parallelparallelize_module)GenerationConfig)PromptFormatter) AutoTokenizer)left_collate_vectors) HFHubMixin)maybe_install_lora)configure_optimizers is_frozen)load_pretrained_hfmove_embeddingsetup_speech_encoder) AudioSignal LabelsType LengthsTypeMaskType NeuralType)loggingc seZdZd9fdd ZeddZedefddZedefd d Zedefd d Z edefd dZ ede fddZ edefddZ  d:dededeeeffddZdefddZdedefddZd9ddZd9dd Zdedefd!d"Zd9d#d$Zd9d%d&Zd'ed(efd)d*Zfd+d,Ze   d;d-eeeeejBd.ejd/ejd0edejf d1d2Z d3d4Z!d9d5d6Z"edefd7d8Z#Z$S)<SALMreturnNcst|tsJdt|dt|t||_|jj|_t |jj dd|_ |j d|jgit |jj |jjd|_|jjj|_|jj`t|t||jjdd|_d|_dS)NzYou must pass the config to SALM as a Python dict to support hyperparameter serialization in PTL checkpoints (we got: 'type(cfg)=z').T)use_fastadditional_special_tokens)pretrained_weightsF) isinstancedicttypesuper__init__save_hyperparametersr cfgaudio_locator_tagrpretrained_llm tokenizeradd_special_tokensrr*llmmodel embed_tokensrr _use_fsdp_use_tp)selfr1 __class__c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/speechlm2/models/salm.pyr/2s$      z SALM.__init__cC|jjS)z&Return the size of the text tokenizer.)r8num_embeddingsr;r>r>r?text_vocab_sizeKszSALM.text_vocab_sizecCr@N)r4bos_idrBr>r>r? text_bos_idPzSALM.text_bos_idcCr@rD)r4eos_idrBr>r>r? text_eos_idTrGzSALM.text_eos_idcCs2|jj}|dur |jj}|durtdd}|S)Nzqthe text tokenizer has no or tokens available, using id 0 for padding (this may lead to silent bugs).r)r4padunk_idwarningswarn)r;pad_idr>r>r? text_pad_idXszSALM.text_pad_idcCs|j|jSrD)r4 token_to_idr2rBr>r>r?audio_locator_tag_iddszSALM.audio_locator_tag_idcCr@)zx Returns the audio duration corresponding to a single frame/token at the output of ``self.perception``. ) perceptiontoken_equivalent_durationrBr>r>r?rShszSALM.token_equivalent_durationcCs |jjjjSrD)rR preprocessor featurizer sample_raterBr>r>r? sampling_rateos zSALM.sampling_rate input_embedsattention_maskcCs<|j||||dudd}d|di}|dur|d|d<|S)z Implements a fully offline forward pass through the entire model. The flow is the following: |speech and text embeddings| -> |llm| -> |lm_head| -> |token ids| NT) inputs_embedsrYpast_key_values use_cache return_dictlogitsr[cache)r6)r;rXrYr_outansr>r>r?forwardss  z SALM.forwardbatchc Cs>|j|d|dd\}}ddt||D}t|d|jkd|d}||}t|d||j|j||d|dd d \}}}|d d d d f}|d d d d f}|d d d d f}|jr|j d } |j d d | } dkr|d d d | f}|d d d | f}|d d d | f}|||dS)a Performs additional processing on the mini-batch collected from dataloader. Notably: * Convert source audio to speech representations. * Convert target audio to target audio tokens. * Convert target text to embeddings. * Combine the input audio and target text embeddings. * Take care of any necessary slicing to align the shapes of source audio, target audio, and target token ids. audios audio_lens) input_signalinput_signal_lengthcSsg|] \}}|d|qSrDr>).0embemblenr>r>r? z'SALM.prepare_inputs.. input_idsr loss_maskrmembeds padding_idplaceholder_id replacements target_idsNtensor_parallel)rXrYru) rRziptorchwhererQr8&replace_placeholders_and_build_targetsrOr: device_meshsizeshape) r;rc audio_embsaudio_emb_lensinput_ids_to_embed text_embs input_embsrurY tp_world_size remainderr>r>r?prepare_inputss6    zSALM.prepare_inputs batch_idxc CsD|jj|jj|jfD] }t|r|q ||}||d|dd}|ddk}t !t j j j |ddd|dddd dd |}Wdn1sUwY|djdd \}} |t |jdurx|jjdjdd nd|| |t j||| |d |jk|d d} |j| dd| S)NrXrYrYruror^rrwsum reduction ignore_indexlrrm)loss learning_rate batch_sizesequence_length num_framestarget_to_input_ratio padding_ratioT)on_step)rRrTencoderr6revalrlongrrrznn functional cross_entropyflattenr as_tensor_trainertrainer optimizers param_groupstofloat32rOnumellog_dict) r;rcrminputsforward_outputsrrBTrar>r>r? training_steps:  (    zSALM.training_stepcCstt|_tt|_dSrD)rlist_partial_val_losses_partial_accuraciesrBr>r>r?on_validation_epoch_starts zSALM.on_validation_epoch_startcCsg}|jD]\}}t|}|jd||ddd||q|jdt|dddg}|jD]\}}t|}|jd||ddd||q9|jdt|ddd|j|jdS)N val_loss_T)on_epoch sync_distval_lossval_acc_val_acc) ritemsrzstackmeanlogappendrclear)r; val_lossesnamevalsr accuraciesaccsrr>r>r?on_validation_epoch_ends   zSALM.on_validation_epoch_endc Cs|D]\}}|dur q||}||d|dd}|ddk}t!tjjj|d dd|d ddd dd |}Wdn1sMwY|dj d d  d } |d d } | | dk} | | dk} | | } |j|| |j||qdS) NrXrYrruror^rrwrrrvdim)rrrrrrzrrrrargmaxviewreshapeeqfloatrrrr) r;rcrr dataset_batchrrrrpredsrefsaccuracyr>r>r?validation_steps2    zSALM.validation_stepcC|SrD)rrBr>r>r?on_test_epoch_startzSALM.on_test_epoch_startcCrrD)rrBr>r>r?on_test_epoch_endrzSALM.on_test_epoch_endargskwargscOs|j|i|SrD)rr;rrr>r>r? test_stepszSALM.test_stepcs>ttj|i|WddS1swYdSrD)rr.backwardrr<r>r?rs"z SALM.backwardpromptsrdregeneration_configc stt|tjr |}n9t||j|jd}dur&|dur|dus"Jd|\}}t|jj |j t fdd|D|j d |j}|dur||||jkd}||} |||\} fddt| Dt|| |j |jdd \} } } | | d }n ||j k} || d }|durt|j|j|j d }t||jjdi||d |i}Wd|S1swY|S)a Generate LLM answers given text or mixed text+audio prompts. Example 1. High-level API using ``prompts`` to provide both text and audio:: >>> answer_ids = model.generate( ... prompts=[ ... [ ... { ... "role": "user", ... "content": f"Transcribe the following: {model.audio_locator_tag}", ... "audio": ["path/to/audio.wav"], ... } ... ] ... ], ... max_new_tokens=128, ... ) You may also include a ``transformers.GenerationConfig`` object to customize decoding strategy:: >>> answer_ids = model.generate(..., generation_config=GenerationConfig(do_sample=True, num_beams=5)) Example 2. Lower-level API, using ``prompts`` for the text part, and pre-loaded ``audio`` and ``audio_lens`` tensors:: >>> answer_ids = model.generate( ... prompts=[ ... [{"role": "user", "content": f"Transcribe the following: {model.audio_locator_tag}"}], ... [{"role": "user", "content": f"Transcribe the following in Polish: {model.audio_locator_tag}"}], ... ], ... audios=audios, # torch.Tensor, float32, of shape (batch, time) ... audio_lens=audio_lens, # torch.Tensor, int64, of shape (batch,) ... max_new_tokens=128, ... ) Example 3. Lower-level API, using pre-tokenized and pre-formatted ``prompts`` for the text part, and pre-loaded ``audio`` and ``audio_lens`` tensors:: >>> answer_ids = model.generate( ... prompts=prompts, # torch.Tensor, int64, of shape (batch, num_tokens) ... audios=audios, # torch.Tensor, float32, of shape (batch, time) ... audio_lens=audio_lens, # torch.Tensor, int64, of shape (batch,) ... max_new_tokens=128, ... ) Inputs: prompts: batch of prompts Tensor or as list[dict] each in the following format [ # batch example id 0 [{"role": "user"}, "slots": {"message": f"Transcribe the following: {model.audio_locator_tag}"}] # batch example id 1 [{"role": "user"}, "slots": {"message": f"Transcribe the following in Polish: {model.audio_locator_tag}"}] ] "role" is LLM-specific, you can pass multiple turns as well. If ``prompts`` is a Tensor, we assume it was already formatted in the relevant chat template and tokenized with the model's tokenizer. audios: Optional. Time-domain audio signal zero-padded batch of shape (B, T). The number of audios must correspond to the number of occurrences of in prompts. Each prompt can have multiple audios. audio_lens: Optional. Length of each audio example. generation_config: Optional HuggingFace GenerationConfig object. generation_kwargs: Keyword arguments passed directly to the underlying LLM's ``generate`` method. )rWdeviceNzaAudios cannot be provided via ``prompts`` and ``audios``/``audio_lens`` arguments simultaneously.csg|] }j|ddqS))turnsrm) encode_dialog)rhprompt) formatterr>r?rkurlz!SALM.generate..) padding_valuercs g|] \}}|d|fqSrDr>)rhielen) audio_embedsr>r?rks rp)rZrY)rmrY) bos_token_id eos_token_id pad_token_idrr>)r+rzr _resolve_audios_in_promptrWrrresolver1 prompt_formatr4rrOrr{rQr8rR enumerater|rrFrIrr6generate)r;rrdrergeneration_kwargstokens maybe_audiotokens_to_embed token_embedsaudio_embed_lensrX_rYgeneration_inputs answer_tokensr>)rrr?r sd I       z SALM.generatecCst|SrD)rrBr>r>r?rrzSALM.configure_optimizersc Cs|j}|dur dS|j}t|tr|jj}|d}dkrd|_tt ft dfddt d}t ||||jj D]_}t ttttt ddt tt dft fdtttt ddd }|j}d D]*}t||}||d krtd |d |d |dt||||qjt |||qr>r?configure_modelsx       zSALM.configure_modelcCsVtdtdtdddtdtdddtdtd|jd d tdtddgd S) z Return a typing schema for optimal batch size calibration for various sequence lengths using OOMptimizer. rd)rrinput)rr- seq_lengthre)rrmoutput)rr-r vocab_sizern)clsr)r,r$r r"r!rCr#rBr>r>r?oomptimizer_schemas zSALM.oomptimizer_schema)r'N)NN)NNN)%__name__ __module__ __qualname__r/propertyrCintrFrIrOrQrrSrWr r,strrbrrrrrrrrrrrzno_gradrrrrrr __classcell__r>r>r<r?r&1sj    3 "     ~ `r&rmrqrrrsrtrur'c Cs|\}}|dur||ksJd|d}|j|j} } d} t||||\}}}g} g} g}d}t|D] }|||kjddd}t|dkrw| |||durm||}| ||||k<| |||||kq:g}g}g}d}|D]i}||kr||||||dur||||}| |||k<||||||||k||}|||t j |df| t j | d|t j |dft j | d|d 7}|d }q||kr"||||||dur||||}| |||k<||||||||k| t j|dd |t j|dd |durE| t j|dd q:|t|krYtd t|d |td d| D}t j|||| | d}|dur~t j ||f| t j | d}nd}t j||ft j | d}|durtd} tt| | |D]-\}\}}}|d}|||| df<|dur|||| df<|||| df<q|||fS)aReplaces each occurrence of the placeholder_id in input_ids with the corresponding tensor from the replacements list in the embeds tensor, and creates corresponding adjusted target_ids. Note: when padding is necessary, we apply left-padding to the examples not to introduce anomalies at generation time. Args: input_ids (Tensor): shape (batch, sequence_length); input token ids. embeds (Tensor): shape (batch, sequence_length, hidden_dim); embeddings for each token. padding_id (int): these IDs will be marked as ignore_index in target_ids. placeholder_id (int): an id to be replaced. replacements (list of Tensor): each Tensor has shape (L_i, hidden_dim), with L_i arbitrary. target_ids (Tensor): shape (batch, sequence_length); target token ids. Returns: Tuple[Tensor, Tensor, Tensor]: - Tensor of shape (batch, max_new_sequence_length, hidden_dim) corresponding to ``embeds`` after replacements. - Tensor of shape (batch, max_new_sequence_length) with adjusted target IDs where: * Original target values are preserved where input was not a placeholder or padding * Positions that were placeholders, padding, or added by replacements are set to -100 Will be None if target_ids input was None. - Tensor of shape (batch, max_new_sequence_length) with attention padding masks updated to account for shape changes due to replacements. Nz0target_ids must have the same shape as input_idsrrorTas_tuple)dtyperrwrz Expected z replacements but used css|]}|dVqdS)rN)r~)rhseqr>r>r? sz9replace_placeholders_and_build_targets..)rr&)r~rr& _unpad_inputsrangenonzerolenrclonerzfullronesboolcat ValueErrormaxzerosrrry) rmrqrrrsrtrurseq_len hidden_dimrr&routput_sequencesoutput_target_idsoutput_att_masksreplacement_idxrplaceholder_positionsnew_target_idssegmentstarget_segments att_masksprev_pospossegment_target_idsrepmax_seq_lengthrattention_masksr'tgtattr>r>r?r|s !       "            r|c Csdd}gg}}|durgnd}t|jdD].}||||} |||| df|||| df|durF|||| dfq|||fS)NcSs2||k}tj|dd}|dkr|dSdS)NFr$rrv)rzr+ritem)tensorvaluemaskindicesr>r>r?first_index_not_values   z,_unpad_inputs..first_index_not_valuer)r*rr) rmrqrurrrMinput_ids_unpad embeds_unpadtarget_ids_unpadrrr>r>r?r)s  r)rrWrc sddlmg}|D]1}|D],}d|vr:|d}t|ttfr"|g}|D]}t|ttfs4Jd|||q$qq |s@dStfdd|D}td| |j dd \} } Wdn1sfwYt | j |dd t | j |dd fS) Nr Recordingaudioz(Invalid value under prompt key 'audio': csg|] }|qSr>) from_fileto_cut)rhprQr>r?rksz-_resolve_audios_in_prompt..cpuT)collate) non_blocking) lhotserRr+r!rrrrzrresample load_audiorr) rrWrpaths conversationturn turn_audiorVcutsrSrer>rQr?rs,   rrD)ErL collectionsr itertoolsrpathlibrtypingrrrzrZr lightningr omegaconfr peftr r torch.distributed.fsdpr torch.distributed.tensorr r!torch.distributed.tensor.parallelrrrrrr transformersrnemo.collections.common.promptsr"nemo.collections.common.tokenizersr,nemo.collections.speechlm2.data.salm_datasetr'nemo.collections.speechlm2.parts.hf_hubr%nemo.collections.speechlm2.parts.lorar,nemo.collections.speechlm2.parts.optim_setuprr+nemo.collections.speechlm2.parts.pretrainedrrrnemo.core.neural_typesr r!r"r#r$ nemo.utilsr%r&r rtupler|r)r,r!rrr>r>r>r?s                 p