o ۷i@sdZddlZddlmZmZmZmZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)e'rddl*m+Z+m,Z,ddl-m.Z.nd\Z.Z,Z+e&rddl/m0Z0m1Z1nd\Z1Z0e2e.e,e0e1e+fZ3e"4e5Z6Gdddej7Z8dej9de:dej9fddZ;Gd d!d!Z< "dCd#ej7d$ej9d%ej9d&ej9d'eej9d(e=d)e=fd*d+Z>Gd,d-d-ej7Z?Gd.d/d/ej7Z@Gd0d1d1ej7ZAGd2d3d3ej7ZBGd4d5d5ej7ZCGd6d7d7ej7ZDe!Gd8d9d9eZEe!Gd:d;d;eEZFGdd?Gd@dAdAeEZHgdBZIdS)DzPyTorch Zamba model.N)AnyCallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNcs.eZdZdfdd ZddZddZZS) ZambaRMSNormư>cs&ttt||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__^/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/zamba/modeling_zamba.pyr&@s  zZambaRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r, hidden_states input_dtypevariancer1r1r2forwardHs  zZambaRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler*shaper+r,r1r1r2 extra_reprOszZambaRMSNorm.extra_repr)r$)__name__ __module__ __qualname__r&r?rC __classcell__r1r1r/r2r#?sr#r<n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r<rHbatchnum_key_value_headsslenhead_dimr1r1r2 repeat_kvTs 0rPc @seZdZdZdZejdfddZddZde d e ej ej ffd d Z dd ej d ej de de eeefd e ej ej ff ddZdejfddZdde e d e fddZdS)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc s"||_d|_|j|_d|_|j|j|_|j|_|j |_ |j |_ g|_ g|_ g|_i|_i|_i|_t|jD];}|j tj|j|j |dg7_ |j |j|j |jf}|j tj||dg7_ |j|dkrr|j|q7fddt|jD|_fddt|jD|_dS)NFdevicer6hybridc g|] }tjggdqSrSr(tensor.0_ batch_sizerSr1r2  z4ZambaHybridDynamicCache.__init__..crUrVrXrZr]r1r2r_r`)r6is_compileablelayers_block_typehas_previous_state mamba_expandr-intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr(zerosappend key_cache value_cache)r,configr^r6rSi cache_shaper1r]r2r&ps:   z ZambaHybridDynamicCache.__init__cCs t|jSN)lenrurBr1r1r2__len__s zZambaHybridDynamicCache.__len__ layer_idxrIcCs|j||j|fSrz)rurvr,r}r1r1r2 __getitem__sz#ZambaHybridDynamicCache.__getitem__ key_states value_states cache_kwargscCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nr4rr3dim)rurArvr(cat)r,rrr}rr1r1r2updates   zZambaHybridDynamicCache.updatebeam_idxcCstt|jD]V}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<qdS)zDReorders the cache for beam search, given the selected beam indices.rN) rqr{rurS index_selectr7rvrkrl)r,rr}rSr1r1r2 reorder_caches     z%ZambaHybridDynamicCache.reorder_cachercCs:||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rmr{rurAr~r1r1r2get_seq_lengthsz&ZambaHybridDynamicCache.get_seq_lengthrz)r)rDrErF__doc__rar(float16r&r|intr@Tensorrrdictstrrr LongTensorrrr1r1r1r2rQ`s(    rQmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr3r rr4)rr6)ptrainingr)rPnum_key_value_groupsr(matmul transposerAr functionalsoftmaxr8r7r6rr contiguous) rrrrrrrkwargsrr attn_weights causal_mask attn_outputr1r1r2eager_attention_forwards & rcseZdZdZdedeffdd Zedddd  dd ej ded e ej de e d e e deej e ej e eej ff ddZZS)ZambaAttentionaA Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Adapted from transformers.models.mistral.modeling_mistral.MistralAttention: The input dimension here is attention_hidden_size = 2 * hidden_size, and head_dim = attention_hidden_size // num_heads. The extra factor of 2 comes from the input being the concatenation of original_hidden_states with the output of the previous (mamba) layer (see fig. 2 in https://huggingface.co/papers/2405.16712). Additionally, replaced attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) with attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim/2) rwr}cst||_||_|j|_|j|_|j|j|_ |j |_ |jdd|_ d|_ |j |_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)Nr3TFbias)r%r&rwr}attention_hidden_sizeattention_head_dimrOnum_attention_headsrMrmax_position_embeddingsr is_causalattention_dropoutrLinearq_projk_projv_projr-o_projr,rwr}r/r1r2r&s  zZambaAttention.__init__past_key_valuepast_key_values4.58new_nameversionNr<rrrIcKs|jdd}g|d|jR}|||dd}|||dd} |||dd} |durB|| | |\} } t} |j j dkrPt |j j } | ||| | |f|j s\dn|j |jd|\} } | jg|dR} || } | | fS)Nr4rr3eagerr)rr)rArOrviewrrrrrrw_attn_implementationrrrrrKrr)r,r<r}rrr input_shape hidden_shape query_statesrrattention_interfacerrr1r1r2r?s2     zZambaAttention.forwardrz)rDrErFrrrr&rr(rrrQrrr@r?rGr1r1r/r2rs$ rcs^eZdZdZdeffdd Z ddejdefdd Z ddefd d Z ddefd d Z Z S)ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rwcst||_||_|j|_|j|_|j|_|j |j|_ |j |_ |j |_ |j |j |_|j|_|j|_tj|j |j |j|j|j |jdd|_|j|_t|j|_|j|_tj|j|j d|jd|_tt |j |j |jd|j|_!tt |j |j|j dd|j d|_"tt |j |j|_#tj$d|jdtj%ddddf}|&|j d'}tt(|)|j |jd|_*tt+|j |j|_,tj|j |j|jd|_-t.st/0ddSdS) Nr) in_channels out_channelsr kernel_sizegroupspaddingr3rg?r6r4aqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r%r&rwr}r-rfrgrhrirdre mamba_dt_ranktime_step_rankrjmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr'r(rs x_proj_weightdt_proj_weight dt_proj_biasaranger8rJrlogrKA_logr)Dout_projis_fast_path_availablelogger warning_once)r,rwr}Ar/r1r2r&.sb  $ zZambaMambaMixer.__init__Nr< cache_paramsc Cs|j\}}}|duo|jo|dk}||dd}||dd|jddd\}} |d}| d} | ||j d|dd} |j j |j j d|j j d} |rnt |d|j|j| |j j|j}|d}nK|durt|dks||d}|durtj||j|jddf} |j|j| t|| |j j|jd}|durt|dks||d}|d|j |j|dd}|jdddddddf|dd} tj| |j|j|jgdd\} }}|j dddf| dd}t!|j"# }|j$dur|j$#nd}tj%|d|f|j&|j'd}|rtt(|j D]K}t)|j*|jdd|f||d df||d df||||dddf||dddf|j+|| |d df||d d d}tj,||fdd}q'nntj%|d|j|jf|j&|j'd}t(|j D]E}t-||||||||dd||dd|j+|#| |||d d d \}}tj,||fdd}tj,||dfdd}q|dur|dur|j*|j||.|dd}|S) Nrr3r4rr)rrrR.T) dt_softplus)delta_softplusreturn_last_state)/rArcrrrchunksqueezerrKrjrr*sizer!rkr}rr unsqueezer(allrrpadricopy_r rrsplitrrgrexprfloatremptyrSr6rqrrlrrrr)r,r<rrr^seq_lenr\use_precomputed_statesprojected_statesgate conv_weightsrkssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statesr1r1r2cuda_kernels_forwardks  $  *         z$ZambaMambaMixer.cuda_kernels_forwardc Cs|j\}}}|j}||dd}||dd|jddd\} } | d} | d} | ||j d|dd} t |t } | r+|j |j jd|kr+|jrZ|j |j } n|j |j } | | j} |jr|dkr|j|j jd|kr|j|j } tj| ddd} | dddddf| dddddf<| |j|j <tj| |jjdddddfdd} |jr| |jj7} || |d} n|durt|dks| |dd| jd dfd} tj | |j!| jddf} | |j|j <||| dd|f} |dur*t|dks*| |dd| jd dfd} nFtj"||j |j#|j$f| j|d} |durOt|dksO| |d} ||| dd|f} |durqt|dksq| |d} | d|j |j#|dd} |j%dddddddf| dd }tj&||j'|j$|j$gdd\}}}|j(dddf|dd |j)ddddddf}tj*|}t+|j,- }t+|ddddddddf|dddddddddf}|dddddddddf|dddddddddf-}|| dddddddddf-}g}t.|D]\}|dddddd|ddfdd| |dddddd|ddfdd} t/| dd||dddd|ddfd}|0|dddddddfq2tj1|dd}|| |j2ddddddf}||| }| r| |j |j <|3|dd|d|dd}|S) Nrr3r4rr)shiftsdims.rRr)4rAr6rrrrrrrKrj isinstancerQrlr}rcloner7rSrcrkr(rollsumrr*rrrrrrrrrirsrrgrrrrrsoftplusrrrrqrrtstackrr)r, input_statesrrr^rr\r6rr<r use_cacher conv_staterrrrrr discrete_A discrete_BdeltaB_urrx scan_outputrr1r1r2 slow_forwards      ( ( & &*  FH* X8&" zZambaMambaMixer.slow_forwardcCs@|jrtr d|jjjvrtd|j|||dS|j|||dS)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrrStype ValueErrorrr)r,r<rrr1r1r2r?)szZambaMambaMixer.forwardr") rDrErFrrr&r(rrQrrr?rGr1r1r/r2r!s > a]rcs$eZdZfddZddZZS)ZambaMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r%r&rwr-rerr gate_projup_proj down_projr hidden_actact_fnr,rwr/r1r2r&7s zZambaMLP.__init__cCs$||||||}|Srz)rr!rr)r,xrr1r1r2r?As zZambaMLP.forward)rDrErFr&r?rGr1r1r/r2r6s  rcseZdZddedeeffdd Zedddd    dd ej d ej ded eej dee dee dee de e deejeeejejfffddZZS)ZambaAttentionDecoderLayerNrwr}csHtt|||_t||_t|j|jd|_ t|j |jd|_ dS)Nr.) r%r&r self_attnr feed_forwardr#r rms_norm_epsinput_layernormr-pre_ff_layernormrr/r1r2r&Gs   z#ZambaAttentionDecoderLayer.__init__rrrrFr<original_hidden_statesroutput_attentionsrrrIc Ksjtj||gdd}||}|jd||||||d|\}} ||}||}|f} |r3| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r4r)r<r}rrr,rNr1)r( concatenater)r&r*r') r,r<r+r}rrr,rrself_attn_weightsoutputsr1r1r2r?Os$   z"ZambaAttentionDecoderLayer.forwardrz)NNFF)rDrErFrrrr&rr(rrQboolrrr@ FloatTensorr?rGr1r1r/r2r$Fs4   r$cseZdZdedeffdd Zedddd dd ejd e ejde ed e ejde ejde e de e de e de ej de ejde eje e ejejfffddZZS)ZambaMambaDecoderLayerrwr}cs4tt||d|_t|j|jd|_||_dS)N)rwr}r%) r%r&rmambar#r-r(r)r}rr/r1r2r&s  zZambaMambaDecoderLayer.__init__rrrrNFr<r+rrr,rcache_positiontransformer_hidden_statesrIc Ksd|} | dur || n|}||}|j|||d}d} | |}|f}|r)|| f7}|r0||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r<rr)r)r3)r,r<r+r}rrrr,rr4r5rresidualr.r/r1r1r2r?s"   zZambaMambaDecoderLayer.forward) NNNNNFFNN)rDrErFrrr&rr(rrrQr0rr@r1r?rGr1r1r/r2r2sF     r2cseZdZdedejdeffdd Zedddd  dd e j d e e j de e de e j de e j de e de ede ede e jdee je ee je jfffddZZS)ZambaHybridLayer shared_transflinearr3cs t||_||_||_dSrz)r%r&r8r9 mamba_decoder)r,r8r9r3r/r1r2r&s  zZambaHybridLayer.__init__rrrrNFr<r+r}rrr,rr4rIc Csp|j|||||||| d} | d} |r| d} || } |j|| ||||| d} |r6| d| f| dd} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r+r}rrr,rr4rr)r5rrr,rr4r3N)r8r9r:) r,r<r+r}rrrr,rr4 layer_outputsr5r.r1r1r2r?s4    zZambaHybridLayer.forward)NNNNNFFN)rDrErFr$rrr2r&rr(rrrrQr0rr@r1r?rGr1r1r/r2r7s@    r7c@s>eZdZUeed<dZdZddgZdZdZ dZ dZ dd Z d S) ZambaPreTrainedModelrwmodelTr$r2rFcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rF|jjjd|d|j durD|jj|j  dSdSt|t rT|jjddSt|tr|jjjd|d|jjd}tj|j| ||jj|jj|jj}tt|jj|t|jjt|jjt|jjj|jj d}|tt!|  }|j"j#|tj$d|j%dtj&ddddf}|'|j(d)}|j*j#t|+|j|j,d|j-jddSdS) Nr)r:stdg?r)minrrr4).rwinitializer_ranger rrrr*datanormal_rzero_ Embedding padding_idxr#fill_rrrinituniform_rrdr-rjr(rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrgr8rJrerrrKrr)r,rr> dt_init_stdrdtinv_dtrr1r1r2 _init_weights sD       $"z"ZambaPreTrainedModel._init_weightsN) rDrErFr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrSr1r1r1r2r<s  r<cseZdZdZdeffdd Ze          ddeej deej deej d ee d eej d ee d ee d ee dee deej deeeffddZddZZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rwc st||j|_|j|_t|j|j|j|_t |}g}g}|j |_ t |j D]2}|j |dkr=| t||dq*|j |dkr\| tj|jj|jjdd| t||dq*t|}t|}g}g|_t|j D]6\}}|dkrd|dgd} g|jfd d | D|_| t|t|t|qo| t|qot||_|j|_t|j|jd |_d|_|dS) Nr3)r}rTFrzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightcsg|]}|qSr1r1)r[r prefix_namer1r2r_lsz'ZambaModel.__init__..r%)r%r& pad_token_idrE vocab_sizerrDr- embed_tokensr$rbrqrrrtr2rrwiter_tied_weights_keys enumerater7next ModuleListlayersrr#r(final_layernormgradient_checkpointing post_init) r,rwblock mamba_layers linear_layersrxrhlayer_id layer_type tied_keysr/r^r2r&Js>     zZambaModel.__init__N input_idsr position_idsr inputs_embedsrr,output_hidden_states return_dictr4rIc  Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}|} t |} |r^|dur^t d| durmt j | jd| jd} |durv| d}|||| } |rdnd}|rdnd}t|jD]A\}}|r|| f7}|jr|jr||j| | ||| |||| }n || | ||| |||| d }|d} |r|ddur||df7}q|| } |r|| f7}|r|jsd |_t| |r|nd||d }| r|S|S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrWrr1)r+r}rrrr,rr4T)last_hidden_staterr< attentions)rwr,ruruse_return_dictrrjrrrrbr(r rrArSr_update_causal_maskrerh_gradient_checkpointing_func__call__rircrto_tuple)r,rrrrsrrtrr,rurvr4r<r+rall_hidden_statesall_self_attnsr}layerr;outputr1r1r2r?ys               zZambaModel.forwardc Csv|jjdkr|durd|vr|SdS|j|j}}t|j}|jd}|dd}tj||f|||d} |dkr@tj | dd} | tj ||d| ddk9} | ddddddf |jdddd} |dur| } |d kr|jd} | d d| fd|ddddddfd} | d d| f| || d d| f<|jjd kr|dur|jjd vrt| |} | S) Nflash_attention_2rrr4) fill_valuer6rS)diagonalrWrr3.sdpa)rxpunpu)rwrr6rSr(finfor?rAfulltriurrKrJr req masked_fillrr_unmask_unattended) r,r input_tensorr4r6rS min_dtypesequence_length target_lengthr mask_length padding_maskr1r1r2rzs0    *  4$   zZambaModel._update_causal_mask NNNNNNNNNN)rDrErFrrr&rrr(rrrQr1r0rr@rr?rzrGr1r1r/r2r\AsL/    nr\cseZdZdeffdd Ze            ddeejdeej deejd ee d eej d eejd ee d ee dee dee deejde eej fde eeffddZ      dddZZS)ZambaForCausalLMrwcsPt|t||_dg|jj|_|j|_tj|j|jdd|_ | dS)Nzlm_head.weightFr) r%r&r\r=rdrarrr-lm_headrkr"r/r1r2r&s   zZambaForCausalLM.__init__Nrrrrrsrrtlabelsrr,rurvr4logits_to_keeprIc  Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} |j|||||||| | | d }|d}t| tr>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrrsrrtrr,rur4rvrrlosslogitsrr<rx)rwr,ruryr=r rslicer loss_functionrarrr<rx)r,rrrrsrrtrrr,rurvr4rrr/r< slice_indicesrrrr1r1r2r?s@( zZambaForCausalLM.forwardTc Ks<|du} | s5|dus|d|jdkr"|dd|jd df}n!|jd|jdkr4|dd|f}nt|j|jd|j|jd}|durl|durl|dd}||dkd| sl|dd|jd df}|durw| rwd|i} nd|i} | |||||jj |d| D] \} } | | vr| | | <q| S)Nr4rr)r6rSrtrr)rsrrrrr4) rArQrwr6rSlongcumsum masked_fill_rrnum_logits_to_keepitems) r,rrrrrtr4rsrr empty_past_kv model_inputsrrr1r1r2prepare_inputs_for_generationjsB     z.ZambaForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNr)NNNNNT)rDrErFrr&rrr(rrrQr1r0rrr@rr?rrGr1r1r/r2r sb        Tra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcseZdZfddZe          ddeejdeejdeejdee e e ej fdeej d eejd ee d ee d ee d ee de eeffddZZS)ZambaForSequenceClassificationcsJt||j|_t||_|jj|_tj|j|jdd|_ | dSr) r%r& num_labelsr\r=rdrrr-scorerkr"r/r1r2r&s    z'ZambaForSequenceClassification.__init__Nrrrrsrrtrrr,rurvrIc  CsB| dur| n|jj} |j|||||||| | d } | d} || } |dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n1|durl||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j || jd |f}d}|dur|| j}|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrsrrtrr,rurvrrz=Cannot handle batch sizes > 1 if no padding token is defined.r4rRz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rW regressionsingle_label_classificationmulti_label_classificationr)rwryr=rrAr`rr7rSr(int32rargmaxrrr0rD problem_typerr6rrr rrrrrrr<rx)r,rrrrsrrtrrr,rurvtransformer_outputsr<rr^last_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrr1r1r2r?sx         "       z&ZambaForSequenceClassification.forwardr)rDrErFr&rrr(rrrr listr1r0r@rr?rGr1r1r/r2rsH      r)rrr\r<)r)JrrJtypingrrrrr(rtorch.nnrrr activationsr cache_utilsr generationr modeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.deprecationrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1dr r!rr get_loggerrDrModuler#rrrPrQrrrrrr$r2r7r<r\rr__all__r1r1r1r2s             g G@EI,L  j