o ei@s0ddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0e1e2Z3eeddGdddeZ4Gddde)Z5Gddde*Z6Gddde'Z7Gdd d e$Z8Gd!d"d"e%Z9ed#deGd$d%d%eZ:eGd&d'd'e(e:Z;Gd(d)d)ej<Z=ed*dGd+d,d,e&e Z>Gd-d.d.ej<Z?eGd/d0d0e(Z@ed1dGd2d3d3e:e0ZAgd4ZBdS)5) dataclassN)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging)merge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introc@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdfdBed<dZ e ejdfdBed<dZejdBed <dZejdBed <dZe dBed <dZe ejdfdBed <dZe ejdfdBed <dZejdBed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r$torch FloatTensor__annotations__r%r&rr'tupler(r)r*r+r,r-r.r7r7a/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/csm/modular_csm.pyr#1s r#c@ eZdZdS) CsmRMSNormNr/r0r1r7r7r7r8r:br:c@r9)CsmRotaryEmbeddingNr;r7r7r7r8r=fr<r=c@r9)CsmMLPNr;r7r7r7r8r>jr<r>c@r9) CsmAttentionNr;r7r7r7r8r?nr<r?c@r9)CsmDecoderLayerNr;r7r7r7r8r@rr<r@z[ The bare Csm Model outputting raw hidden-states without any specific head on top. cs`eZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ eedZefdd ZZS) CsmPreTrainedModelconfigmodel)audiotextTr@r&)r'r(cszt|t|tr$|j}t|dD] }tj|jd|j j dqdSt|t r;t |j t|j j|j jdSdS)Nrg)meanstd)super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeinitnormal_weightrBinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr3arange vocab_size)selfmodulerLi __class__r7r8rIs   $z CsmPreTrainedModel._init_weights)r/r0r1rr5base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendr@r?_can_record_outputsr3no_gradrI __classcell__r7r7rZr8rAvs rAcseZdZUeed<fddZeee        dde j dBde j dBde j dBde j dBd e dBd e j dBd edBd e j dBd eedeeBfddZZS)CsmDepthDecoderModelrBcs>t|t|j|j|j|_tj|j|j dd|_ dSNF)bias) rH__init__nn EmbeddingrLrVbackbone_hidden_size embed_tokensLinear hidden_sizeinputs_embeds_projectorrWrBrZr7r8rks zCsmDepthDecoderModel.__init__N input_idsbackbone_last_hidden_stateattention_mask position_idsr& inputs_embeds use_cachecache_positionkwargsreturnc  Ks|durtstdd}|du|duArtd|r&|dur&t|jd}|durV|dur2|nd} |dur=|jdn|jd} |durI|jn|j} t j | | | | d}|durt j |ddd} | |j }| ||}|ddk}|dur||dddf<n ts|rtd ||}t|j|||||d }|}|d}|j||d }|jd|jjD]}||f||||||d | }q||}t||r|d Sdd S)aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)rBrrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rBrxrvrzr&rw)rw)rvrwr&ryrzposition_embeddings)last_hidden_stater&)rlogger warning_once ValueErrorrrBget_seq_lengthshaper~r3rUclamprVrowarningrrr unsqueeze rotary_emblayersnum_hidden_layersnormr )rWrtrurvrwr&rxryrzr{past_seen_tokensinputs_seq_lengthr~ codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr'r decoder_layerr7r7r8forwardsr        zCsmDepthDecoderModel.forward)NNNNNNNN)r/r0r1r r5rkrrrr3 LongTensorr4Tensorrboolr rr6r rrgr7r7rZr8rhsF     rhcs&eZdZfddZdddZZS)rKcs0t||_tt|jd|||_dS)Nr)rHrkrLrl Parameterr3emptyrP)rWrqrLrVrZr7r8rks  zCsmCodebooksHead.__init__Ncsf|durjd}|jt|n |d}|j|fddtjdDtjddS)Nrc s2g|]}tjdd|ddf|jqSN)rl functionallinearT).0 codebook_idxcodebook_weightr'r7r8 s$z,CsmCodebooksHead.forward..rdim)rrPr3rUrMstack)rWr'rz seq_lengthrr7rr8rs    zCsmCodebooksHead.forwardrr/r0r1rkrrgr7r7rZr8rKs rKa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) cseZdZdZdZdZfddZ    ddejde dBdejdBdej dBdejdBf fd d Z e e          ddejdBd ej dBdejdBd ejdBde dBdej dBdejdBdedBdejdBdeejBdeedeeBfddZZS)CsmDepthDecoderForCausalLMNcs2t||`t|j|j|j|_t||_ dSr) rHrklm_headrKrqrLrVcodebooks_headrhrCrsrZr7r8rks z#CsmDepthDecoderForCausalLM.__init__rtr&rvrxrzc  sHtj|||||fi|}|dddk}|s|d|d|S)Nrzrrurw)rHprepare_inputs_for_generationpop) rWrtr&rvrxrzr{ model_inputsis_first_generation_steprZr7r8r$s   z8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationrrurwlabelsrylogits_to_keepr{r|c  Ks|jd|||||||| d| } | d} t| tr+| dkr$tdd}n t| d}n| }|| dd|ddf| durA| |nd}|}d}|durg|dddf}|jd|d|jj|d| }t ||| j | j | j dS) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rtrurvrwr&rxryrzrrN.)r%rrV shift_labels)r$r%r&r'r(r7) rCrJintslicer contiguous loss_functionrBrVr r&r'r()rWrtrurvrwr&rxrryrzrr{outputsr' slice_indicesr%r$rr7r7r8r:sJ   & z"CsmDepthDecoderForCausalLM.forwardNNNN) NNNNNNNNNr)r/r0r1_tied_weights_keys_tp_plan_pp_planrkr3rrr4rrrrrrr rr6r rrgr7r7rZr8rsr       rcs$eZdZfddZddZZS)rRcsDtt|j|j|j|_|jdt |j|jdddS)NrTF) persistent) rHrkrlrmrLrVrqembed_audio_tokensregister_bufferr3rUrsrZr7r8rks  z#CsmBackboneModelEmbeddings.__init__cCs |||j}|jdd}|S)Nrr)rrTsum)rWrtrxr7r7r8rs z"CsmBackboneModelEmbeddings.forwardrr7r7rZr8rRs rRcs4eZdZfddZeeefddZZS)CsmBackboneModelcst|t||_dSr)rHrkrRrorsrZr7r8rks zCsmBackboneModel.__init__c stjdi|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nr7)rHr)rW super_kwargsrZr7r8rszCsmBackboneModel.forward) r/r0r1rkrrrrrgr7r7rZr8rs  rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. cs~eZdZddiZfddZddZddZefd d Zfd d Z d"de j d Bde j d Bde j d Bde j d Bde j d Bf ddZ d"de j ded Bde j d Bde jd Bde j d Bf fdd Zee d#de j d Bde j d Bde j d Bde j d Bde j d Bded Bde jd Bde j d Bded Bde j d Bdee j BdeedeeBfd d!ZZS)$CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightcspt||j|_tj|j|jdd|_t|j|j|_ t ||_ t |j|_t|j|_|dSri)rHrkrVrlrprqrrmtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrsrZr7r8rks   z$CsmForConditionalGeneration.__init__cCs|jjSrrro)rWr7r7r8get_input_embeddingssz0CsmForConditionalGeneration.get_input_embeddingscCs ||j_dSrr)rWvaluer7r7r8set_input_embeddingss z0CsmForConditionalGeneration.set_input_embeddingscs|ddrtj|i|\}}n tj|i|}dtfddt|jD}t|jjddi||D] }t |j|q?d|vrR||fS|S)Noutput_loading_infoFdepth_decoder_cs(i|]\}}|r|d|qSr) startswith)rattrrprefix prefix_lenr7r8 s  z?CsmForConditionalGeneration.from_pretrained.._from_model_config) getrHfrom_pretrainedlenvarsgeneration_configitemsrupdatedelattr)clsargsr{rC loading_infodepth_decoder_attrsrrZrr8rs   z+CsmForConditionalGeneration.from_pretrainedcsVd}|jj}|dd|D] \}}t|j|||qtj|i|dS)Nrtransformers_version)rr to_diff_dictrrsetattrrHsave_pretrained)rWrr{rrrrrZr7r8rs   z+CsmForConditionalGeneration.save_pretrainedNrt input_valuesinput_values_cutoffsrr|csF||}|durtj|d}||dk}||dk}tj||jd t |d}|| dk}t jg}t ||D]?\} } | | dk} t| jddD]+} | | } | | d} | d| | f}|j| d}|jdd}||dqUqBtdd |Dtfd d |D}|j|}Wdn1swY|jj}||k}|j|}||||<tjdd|jjf|jtjd |jj}|j|d}||jj k}|!|"d||<|dur| d!dd|jj}||||<|||<|d kj#dd}d||d|dddf<|}||dS)a Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retrieve codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrr}r.css|]}|jdVqdS)rN)rrelr7r7r8 szQCsmForConditionalGeneration._merge_input_ids_with_input_values..c s,g|]}tj|ddd|jdfqS)r)rlrpadrrmax_audio_framesr7r8rs,zRCsmForConditionalGeneration._merge_input_ids_with_input_values..)r~dtypeiTas_tuple)rxr)$rrlrrdiffr3rUmaxr~expandrrrfziprMrrencode audio_codes transposeappendrget_audio_codes_maskrBaudio_token_idrroonesrLlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rWrtrrrrx audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrY start_idxend_idx audio_batch codec_outputs codebook_idsbatched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxsr7rr8"_merge_input_ids_with_input_valuess\            z>CsmForConditionalGeneration._merge_input_ids_with_input_valuesr&rvrxrzc  stjd |||||d|}|dur>|jdkr>|ddur>|j||d|d|dd}||d|ddd|S) N)rtr&rvrxrzrrxrrr)rtrrr)rxrrtr7)rHrndimrrr) rWrtr&rvrxrzr{r merged_inputsrZr7r8r7s(  z9CsmForConditionalGeneration.prepare_inputs_for_generationrrwryrr{c  Ks|dur|jdkr|||||} | d}| d}d}|jd|||||| | d| }|d}t| tr:t| dn| }||dd|ddf}d}d}d}d}|dur|dddddf}|jd|||jj d| }|ddddddfd kj d d }||d d|jj df}t j j|d dd}|jdd}||d|ddddf}||}|jd||| d|d| }|j}||}t|||||j|j|j|dur|jnd|dur|jnd|dur|jnd|dur|jd Sdd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrxr)rtrvrwr&rxryrzr)r%rrVrrrr.r)rTr)rtrury return_dictr) r$r.r)r%r&r'r(r*r+r,r-r7)rrrrJrrrrrBrVallrLrlrrr rr$r#r&r'r(r%)rWrtrrvrrwr&rxrryrzrr{rbackbone_outputsbackbone_hidden_statesrbackbone_logitsr$r.r)depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelsr7r7r8rVsS  (   z#CsmForConditionalGeneration.forwardr) NNNNNNNNNNr)r/r0r1rrkrr classmethodrrr3rrrrr4rrrrrr rr6r#rrgr7r7rZr8rs     U     r)rArrhrr)C dataclassesrr3torch.nnrlrrN cache_utilsrr generationr masking_utilsrmodeling_outputsr r modeling_utilsr processing_utilsr utilsr rrr utils.genericrutils.import_utilsrutils.output_capturingrautorllama.modeling_llamarrrrrrrrconfiguration_csmrr generation_csmr! get_loggerr/rr#r:r=r>r?r@rArhModulerKrrRrr__all__r7r7r7r8sf           (  +`f M