o eii@s\ddlZddlmZddlmZddlZddlmmZ ddlmZddl m Z ddl mZddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZddlmZmZddl m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3edGdddej4Z5Gdddej4Z6Gdddej4Z7ddZ8ed dBd!d"Z9d#ej:d$e;d%ej:fd&d'Z< (dCd)ej4d*ej:d+ej:d,ej:d-ej:dBd.e=d/e=d0e'e)fd1d2Z>dDd3d4Z?dEd5d6Z@Gd7d8d8ej4ZAGd9d:d:eZBe*Gd;d<dd>eCZDe*Gd?d@d@eCeZEgdAZFdS)FN)Callable)Optional)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)is_flash_attention_requestedmaybe_autocastmerge_with_config_defaults)capture_outputs) YoutuConfigRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) YoutuRMSNormư>epsreturnNcs&ttt||_||_dS)z; YoutuRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer# __class__f/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/youtu/modeling_youtu.pyr&6s  zYoutuRMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r,r2 input_dtypevariancer0r0r1forward>s  zYoutuRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler*shaper+)r,r0r0r1 extra_reprEszYoutuRMSNorm.extra_repr)r") __name__ __module__ __qualname__floatr&r(Tensorr>rA __classcell__r0r0r.r1r!4sr!c s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)YoutuRotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrIF) persistentoriginal_inv_freq) r%r&max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrJrope_parametersrKcompute_default_rope_parametersrattention_scalingregister_bufferclone)r,rJdevice rope_init_fnrIr.r0r1r&Ls   zYoutuRotaryEmbedding.__init__rWz torch.deviceseq_lenr$z torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?rr3r6)rWr6) rRgetattrr-num_attention_headsr(arangeint64r7rE)rJrWrYbasedimattention_factorrIr0r0r1rS\s &z4YoutuRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) Nrr4rmpscpuF) device_typeenabledr3rcr])rIrEexpandr@r7rW isinstancetypestrr transposer(catcosrTsinr6) r,x position_idsinv_freq_expandedposition_ids_expandedrgfreqsembrprqr0r0r1r>zs0&zYoutuRotaryEmbedding.forwardN)NNN)rBrCrDr(rF__annotations__rr& staticmethodrintr?rErSno_gradrr>rGr0r0r.r1rHIs&   rHcs$eZdZfddZddZZS)YoutuMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r%r&rJr-intermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr,rJr.r0r1r&s zYoutuMLP.__init__cCs$||||||}|Srx)rrrr)r,rrrr0r0r1r>s zYoutuMLP.forward)rBrCrDr&r>rGr0r0r.r1r}s  r}cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr4r3ri)r@r(ro)rrx1x2r0r0r1 rotate_halfsrrotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrprq unsqueeze_dimq_embedk_embedr0r0r1apply_rotary_pos_embs  rr2n_repr$cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r@rjreshape)r2rbatchnum_key_value_headsslenr[r0r0r1 repeat_kvs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr3rr4)rcr6)ptrainingr)rnum_key_value_groupsr(matmulrnr functionalsoftmaxr8r7r6rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr0r0r1eager_attention_forwards  rc Cs||}||}|j\}}}} ||||| dddd|||| }|j\}}}} ||||| dddd|||| }||t||} ||t||} | | fS)a TODO let's just use the original freqcis computation to not have the view transpose + reshape! This is not optimized! Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r3r)rr@viewrnrr) rrrprqrsrbhsdrrr0r0r1apply_rotary_pos_emb_interleaves  **rcCs"|dkrdSd|t|dS)Nrr\g?)mathlog)scalemscaler0r0r1yarn_get_mscalesrcseZdZdZdedeffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e e d e ejejdBe ejdBffddZZS)YoutuAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrJ layer_idxcst||_||_|j|j|_|j|_|j|_|j |_ |j |_ |j |_ |j |_ |j |_ |j|_d|_|j durItj|j|j|jdd|_n tj|j|j |jd|_t|j |_tj|j |j|jdd|_tj|j|j |j |jd|_t|j |_tj|j |j|j |j dd|_tj|j|j |j|jd|_|jd|_|jjdddkr|jjdd}|jjd }|rt ||}|j|||_dSdSdS) NTFrgrKrLmscale_all_dimrfactor)!r%r&rJrr_rrattention_dropout num_heads q_lora_rankqk_rope_head_dim kv_lora_rank v_head_dimqk_nope_head_dim qk_head_dim is_causalrrr-q_projattention_biasq_a_projr! q_a_layernormq_b_projkv_a_proj_with_mqakv_a_layernorm kv_b_projo_projrrRgetr)r,rJrrscaling_factorrr.r0r1r&sV         zYoutuAttention.__init__Nr2position_embeddingsrpast_key_valuescache_positionrr$cKsp|jdd\}}||d|jf} ||d|j|jf} |jdur%||} n ||||} | |  dd} t j | |j|j gdd\} } ||}t j ||j|j gdd\}}||| |  dd}t j ||j|jgdd\}}| |d||j }|\}}|jjrt| |||\} }n t| |||\} }|jg|jdddR}t j| | fdd}t j||fdd}|dur|||d}||||j|\}}t|jr|j|jkrt|d|j|jg}t|jjt }||||||f|j!sdn|j"|j#d|\}}t|jr&|j|jkr&|ddddddd|jf}|$||d%}|&|}||fS) Nr4rr3ri)rqrprrr)rr)'r@rrrrrrrrrrnr(splitrrrrrrJrope_interleaverrrjroupdaterrFpadr get_interface_attn_implementationrrrrrrr)r,r2rrrrr batch_size seq_length query_shape key_shapeq_statesq_passq_rot compressed_kvk_passk_rotrrprq query_statesr cache_kwargsattention_interfacerrr0r0r1r>@sZ       " zYoutuAttention.forward)NN)rBrCrD__doc__rr{r&r(rFr?r LongTensorrrr>rGr0r0r.r1r s(6rcseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e dejfddZZS)YoutuDecoderLayerrJrcsRt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ dS)N)rJrr#) r%r&r-r self_attnr}mlpr! rms_norm_epsinput_layernormpost_attention_layernorm)r,rJrr.r0r1r&s  zYoutuDecoderLayer.__init__NFr2rrsr use_cacherrrr$c Ks^|} ||}|jd|||||||d|\}} | |}|} ||}||}| |}|S)N)r2rrsrrrrr0)rrrr) r,r2rrsrrrrrresidual_r0r0r1r>s&     zYoutuDecoderLayer.forward)NNNFNN)rBrCrDrr{r&r(rFrrboolr?rrr>rGr0r0r.r1rs6   rcs`eZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZefddZZS) YoutuPreTrainedModelrJmodelTrr)r2 attentionscstt|t|jdd}t|jdd|}t|tjr6tj|j d|d|j dur8t |j j |j dSdSdS)Ninitializer_rangeg{Gz?embedding_initializer_ranger3r)r:std) r% _init_weightsr^rJrkr Embeddinginitnormal_r* padding_idxzeros_data)r,rr embed_stdr.r0r1rs   z"YoutuPreTrainedModel._init_weights)rBrCrDrrybase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr(r|rrGr0r0r.r1rs rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd ej dBd e dBd eed efddZZS) YoutuModelrJcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr0)r).0rrJr0r1 sz'YoutuModel.__init__..rrF)r%r& pad_token_idr vocab_sizerrr- embed_tokens ModuleListrangenum_hidden_layerslayersr!rnormrH rotary_embgradient_checkpointing post_initrr.rr1r&s   zYoutuModel.__init__N input_idsrrsr inputs_embedsrrrr$c Ks|du|duAr td|dur||}|r!|dur!t|jd}|dur<|dur-|nd} tj|jd|jd| }|durE| d}t |j|||||d} |} |j | |d} |j d|jj D]} | | f| | ||||d|} qb|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rW)rJrrrrrs)rs)rrrsrrr)last_hidden_stater) ValueErrorrr rJget_seq_lengthr(r`r@rWrr rrrrr)r,rrrsrrrrrpast_seen_tokens causal_maskr2r decoder_layerr0r0r1r>sP      zYoutuModel.forward)NNNNNNN)rBrCrDrr&rrrr(rrFr FloatTensorrrrrr>rGr0r0r.r1rs>  rcseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bde j d Bdee j BdeedefddZZS)YoutuForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr2logitscs@t|t||_|j|_tj|j|jdd|_| dSr~) r%r&rrrrrr-r(rrr.r0r1r&&s   zYoutuForCausalLM.__init__Nrrrrsrrlabelsrrlogits_to_keeprr$c  Ks|jd|||||||d| } | j} t| trt| dn| } || dd| ddf}d}|durB|jd|||jjd| }t ||| j | j | j dS)a Example: ```python >>> from transformers import AutoTokenizer, YoutuForCausalLM >>> model = YoutuForCausalLM.from_pretrained("meta-youtu/Youtu-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-youtu/Youtu-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rrrsrrrrN)r*r+r)lossr*rr2rr0) rr rkr{slicer( loss_functionrJrrrr2r)r,rrrsrrr+rrr,routputsr2 slice_indicesr*r-r0r0r1r>/s0  zYoutuForCausalLM.forward) NNNNNNNNr)rBrCrD_tied_weights_keys_tp_plan_pp_planr&rrr(rrFrr&rr{rrrr>rGr0r0r.r1r' sN      r')rrr')r)r)Nr)rr)Grcollections.abcrtypingrr(torch.nn.functionalrrrrr activationsr cache_utilsrr generationr integrationsr r masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrrutils.output_capturingrconfiguration_youturModuler!rHr}rrrFr{rrErrrrrrrr'__all__r0r0r0r1sv            A   &y-P K