o wi@sddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZdd lmZd efd d Zdejfd edefddZeddZdejjfddZddejjdefddZdS))contextmanager)PathN) open_dict) PeftModel) AutoConfigAutoModelForCausalLM)ASRModel)AudioPerceptionModule)fp32_precision)AudioCodecModelmodel_path_or_namecCs*t|r|dr||S||S)z Load pretrained NeMo 1.0 model (inheriting from ModelPT). Works with ASR, TTS, codec models. Setting ``pretrained_weights=False`` returns a model that has identical architecture with the checkpoint, but is randomly initialized. z.nemo)rexistsendswith restore_fromfrom_pretrained)clsr rh/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/speechlm2/parts/pretrained.pyload_pretrained_nemos  rTpretrained_weightscCs*|r tj||dSt|}tj||dS)z Load pretrained HuggingFace AutoModelForCausalLM. Setting ``pretrained_weights=False`` returns a model that has identical architecture with the checkpoint, but is randomly initialized. ) torch_dtype)rrr from_config)r rdtypeconfigrrrload_pretrained_hf*s rccsZt|jtr|j|jjjj_n|j|jj_dVt|jtr'|jjjj`dS|jj`dS)zKTemporarily restores the embedding layer into HF LLM. Supports LoRA models.N) isinstancellmr embed_tokens base_modelmodel)rrrrmove_embedding8s    r rcCs~t|drt|jjtjkrdSttt |j j  |_Wdn1s*wY|jD]}d|_ q4|j`dS)z Sets up an ``AudioCodecModel``, initializing it from pretrained weights. The result is assigned to ``model.audio_codec`` attribute. Includes a workaround for PTL auto-downcasting the codec model to bf16 with bf16-true precision. audio_codecNF)hasattrnextr! parametersrtorchfloatr rr cfgpretrained_audio_codeceval requires_grad discriminator)rprrrsetup_audio_codecFs  r-cCs|rKtt|jj}t|j|jj|jj_|jj|jj_|j j j |jj_ Wdn1s1wYt |jj|_|jj|dddSt |jj|_dS)z Sets up an ``AudioPerceptionModule``, initializing its ``encoder`` and ``preprocessor`` with a pretrained NeMo ``ASRModel``. The result is assigned to ``model.perception`` attribute and is trainable. NF)strict)rrr'pretrained_asrr)r preprocessor perceptionencoderrr hidden_size output_dimr trainload_state_dict state_dict)rrasrrrrsetup_speech_encoderVs r9)T) contextlibrpathlibrr% omegaconfrpeftr transformersrrnemo.collections.asr.modelsr"nemo.collections.speechlm2.modulesr *nemo.collections.speechlm2.parts.precisionr nemo.collections.tts.modelsr strrfloat32boolrr nnModuler-r9rrrrs