o eiD@sdZddlZddlmZddlmZddlZddlmZddlm Z m Z m Z ddl m Zdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+e)rddl,m-Z-m.Z.ddl/m0Z0nd\Z0Z.Z-e(rddl1m2Z2m3Z3nd\Z3Z2e4e0e.e2e3e-fZ5e&6e7Z8Gdddej9Z:dej;de $dEd%ej9d&ej;d'ej;d(ej;d)ej;dBd*e?d+e?fd,d-Z@Gd.d/d/ej9ZAGd0d1d1ej9ZBGd2d3d3ej9ZCGd4d5d5ej9ZDGd6d7d7eZEGd8d9d9eZFe%Gd:d;d;e!ZGe%Gdd?d?eGeZIe%d@dAGdBdCdCeGZJgdDZKdS)FzPyTorch Zamba model.N)Callable)Any)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) ZambaRMSNormư>epsreturnNcs&ttt||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer$ __class__f/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/zamba/modeling_zamba.pyr'As  zZambaRMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r-r3 input_dtypevariancer1r1r2forwardIs  zZambaRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler+shaper,r-r1r1r2 extra_reprPszZambaRMSNorm.extra_repr)r#) __name__ __module__ __qualname__floatr'r)Tensorr?rC __classcell__r1r1r/r2r"@sr"r3n_repr%cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r3rJbatchnum_key_value_headsslenhead_dimr1r1r2 repeat_kvUs 0rQc @seZdZdZdZejdfddZddZ ddej d ej d e d e e e fdBd eej ej ff d dZdejfddZdd e dBd e fddZdej d e d ee e ffddZdS)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc s"||_d|_|j|_d|_|j|j|_|j|_|j |_ |j |_ g|_ g|_ g|_i|_i|_i|_t|jD];}|j tj|j|j |dg7_ |j |j|j |jf}|j tj||dg7_ |j|dkrr|j|q7fddt|jD|_fddt|jD|_dS)NFdevicer7hybridc g|] }tjggdqSrTr)tensor.0_ batch_sizerTr1r2  z4ZambaHybridDynamicCache.__init__..crVrWrYr[r^r1r2r`ra)r7is_compileablelayers_block_typehas_previous_state mamba_expandr.intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr)zerosappend key_cache value_cache)r-configr_r7rTi cache_shaper1r^r2r'qs:   z ZambaHybridDynamicCache.__init__cCs t|jSN)lenrvrBr1r1r2__len__s zZambaHybridDynamicCache.__len__ key_states value_states layer_idx cache_kwargsr%cCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nr5rr4dim)rvrArwr)cat)r-r~rrrr1r1r2updates   zZambaHybridDynamicCache.updatebeam_idxcCs|dkrdtt|jD]X}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j |j}|j |d|||j |<q dSdS)zDReorders the cache for beam search, given the selected beam indices.rN) get_seq_lengthrrr|rvrT index_selectr8rwrlrm)r-rrrTr1r1r2 reorder_caches      z%ZambaHybridDynamicCache.reorder_cachercCsN||jvr |jdn|}t|j|ks|j|jddkrdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rr5)rnr|rvrA)r-rr1r1r2rs"z&ZambaHybridDynamicCache.get_seq_lengthcache_positioncCs$d}|jd}|||}||fS)zDReturn the length and offset of the cache, used to generate the maskr)rAr)r-rr kv_offset query_length kv_lengthr1r1r2get_mask_sizess z&ZambaHybridDynamicCache.get_mask_sizesr{)r)rDrErF__doc__rbr)float16r'r}rHintdictstrrr@r LongTensorrrrr1r1r1r2rRas(    $ rRmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr4rr5)rr7)ptrainingr)rQnum_key_value_groupsr)matmul transposer functionalsoftmaxr9r8r7rr contiguous) rrrrrrrkwargsr~r attn_weights attn_outputr1r1r2eager_attention_forwards  rcs|eZdZdZdedeffdd Z ddejdedejdBd e dBd e e d e ejejdBe ejdBff d d Z ZS)ZambaAttentionaA Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Adapted from transformers.models.mistral.modeling_mistral.MistralAttention: The input dimension here is attention_hidden_size = 2 * hidden_size, and head_dim = attention_hidden_size // num_heads. The extra factor of 2 comes from the input being the concatenation of original_hidden_states with the output of the previous (mamba) layer (see fig. 2 in https://huggingface.co/papers/2405.16712). Additionally, replaced attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) with attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim/2) rxrcst||_||_|j|_|j|_|j|j|_ |j |_ |jdd|_ d|_ |j |_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)Nr4TFbias)r&r'rxrattention_hidden_sizeattention_head_dimrPnum_attention_headsrNrmax_position_embeddingsr is_causalattention_dropoutrLinearq_projk_projv_projr.o_projr-rxrr/r1r2r's  zZambaAttention.__init__Nr3rpast_key_valuesrr%cKs|jdd}g|d|jR}|||dd}|||dd} |||dd} |durB|| | |\} } t |j j t } | ||| | |f|j sVdn|j|jd|\} } | jg|dR} || } | | fS)Nr5rr4r)rr)rArPrviewrrrrr get_interfacerx_attn_implementationrrrrrLrr)r-r3rrrr input_shape hidden_shape query_statesr~rattention_interfacerrr1r1r2r?s2  zZambaAttention.forwardr{)rDrErFrrrr'r)rHrRrrr@r?rIr1r1r/r2rs" rcs^eZdZdZdeffdd Z ddejdefdd Z ddefd d Z ddefd d Z Z S)ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rxcst||_||_|j|_|j|_|j|_|j |j|_ |j |_ |j |_ |j |j |_|j|_|j|_tj|j |j |j|j|j |jdd|_|j|_t|j|_|j|_tj|j|j d|jd|_tt |j |j |jd|j|_!tt |j |j|j dd|j d|_"tt |j |j|_#tj$d|jdtj%ddddf}|&|j d'}tt(|)|j |jd|_*tt+|j |j|_,tj|j |j|jd|_-t.st/0ddSdS) Nr) in_channels out_channelsr kernel_sizegroupspaddingr4rg?r7r5aqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r&r'rxrr.rgrhrirjrerf mamba_dt_ranktime_step_rankrkmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr(r)rt x_proj_weightdt_proj_weight dt_proj_biasaranger9rKrlogrLA_logr*Dout_projis_fast_path_availablelogger warning_once)r-rxrAr/r1r2r'3sb  $ zZambaMambaMixer.__init__Nr3 cache_paramsc Cs|j\}}}|duo|jo|dk}||dd}||dd|jddd\}} |d}| d} | ||j d|dd} |j j |j j d|j j d} |rnt |d|j|j| |j j|j}|d}nK|durt|dks||d}|durtj||j|jddf} |j|j| t|| |j j|jd}|durt|dks||d}|d|j |j|dd}|jdddddddf|dd} tj| |j|j|jgdd\} }}|j dddf| dd}t!|j"# }|j$dur|j$#nd}tj%|d|f|j&|j'd}|rtt(|j D]K}t)|j*|jdd|f||d df||d df||||dddf||dddf|j+|| |d df||d d d}tj,||fdd}q'nntj%|d|j|jf|j&|j'd}t(|j D]E}t-||||||||dd||dd|j+|#| |||d d d \}}tj,||fdd}tj,||dfdd}q|dur|dur|j*|j||.|dd}|S) Nrr4r5rr)rrrS.T) dt_softplus)delta_softplusreturn_last_state)/rArdrrrchunksqueezerrLrkrr+sizer rlrrr unsqueezer)allrrpadrjcopy_rrrsplitrrhrexprrGremptyrTr7rrrrmrrrr)r-r3rrr_seq_lenr]use_precomputed_statesprojected_statesgate conv_weightsrlssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statesr1r1r2cuda_kernels_forwardps  $  *         z$ZambaMambaMixer.cuda_kernels_forwardc CsX|j\}}}|j}||dd}||dd|jddd\} } | d} | d} | ||j d|dd} t |t } | r|j |j jd|kr|jrZ|j |j } n|j |j } | | j} |jr|dkr|j|j jd|kr|j|j } tj| ddd} | dddddf| dddddf<| |j|j <tj| |jjdddddfdd} |jr| |jj7} || |d} n|dur| |dd| jd dfd} tj| |j | jddf} | |j|j <||| dd|f} |dur| |dd| jd dfd} n6tj!||j |j"|j#f| j|d} |dur8| |d} ||| dd|f} |durR| |d} | d|j |j"|dd} |j$dddddddf| dd }tj%||j&|j#|j#gdd\}}}|j'dddf|dd |j(ddddddf}tj)|}t*|j+, }t*|ddddddddf|dddddddddf}|dddddddddf|dddddddddf,}|| dddddddddf,}g}t-|D]\}|dddddd|ddfdd| |dddddd|ddfdd} t.| dd||dddd|ddfd}|/|dddddddfqtj0|dd}|| |j1ddddddf}||| }| r| |j |j <|2|dd|d|dd}|S) Nrr4r5rr)shiftsdims.rSr)3rAr7rrrrrrrLrk isinstancerRrmrrcloner8rTrdrlr)rollsumrr+rrrrrrrrjrtrrhrrrrrsoftplusrrrGrrrrustackrr)r- input_statesrrr_rr]r7rr3r use_cacher conv_staterrrrrr discrete_A discrete_BdeltaB_urry scan_outputrr1r1r2 slow_forwards      ( ( &  &  *  FH* X8&" zZambaMambaMixer.slow_forwardcCs@|jrtr d|jjjvrtd|j|||dS|j|||dS)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrrTtype ValueErrorrr)r-r3rrr1r1r2r?.szZambaMambaMixer.forwardr!) rDrErFrrr'r)rHrRrrr?rIr1r1r/r2r&s > a]rcs$eZdZfddZddZZS)ZambaMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r&r'rxr.rfrr gate_projup_proj down_projr hidden_actact_fnr-rxr/r1r2r'<s zZambaMLP.__init__cCs$||||||}|Sr{)rrrr)r-xrr1r1r2r?Fs zZambaMLP.forward)rDrErFr'r?rIr1r1r/r2r;s  rcseZdZddededBffdd Z    ddejdejded ejdBd edBd e dBd e dBd e e de ej e ej ej fdBffddZZS)ZambaAttentionDecoderLayerNrxrcsHtt|||_t||_t|j|jd|_ t|j |jd|_ dS)Nr$) r&r'r self_attnr feed_forwardr"r rms_norm_epsinput_layernormr.pre_ff_layernormrr/r1r2r'Ls   z#ZambaAttentionDecoderLayer.__init__Fr3original_hidden_statesrroutput_attentionsr rr%c Ksjtj||gdd}||}|jd||||||d|\}} ||}||}|f} |r3| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r5r)r3rrrr(r Nr1)r) concatenater%r"r&r#) r-r3r'rrrr(r rself_attn_weightsoutputsr1r1r2r?Ts$   z"ZambaAttentionDecoderLayer.forwardr{)NNFF)rDrErFrrr'r)rHrRboolrrr@ FloatTensorr?rIr1r1r/r2r Ks2   r cseZdZdedeffdd Z          ddejdejdBdedBd ejdBd ejdBd edBd e dBd e dBdej dBdej dBdejdBde ej e ej ej fdBffddZ ZS)ZambaMambaDecoderLayerrxrcs4tt||d|_t|j|jd|_||_dS)N)rxrr!) r&r'rmambar"r.r$r%rrr/r1r2r's  zZambaMambaDecoderLayer.__init__NFr3r'r causal_maskrr(r r position_idstransformer_hidden_statesr%c Ksd|} | dur || n|}||}|j|||d}d}| |}|f}|r)||f7}|r0||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r3rr)r%r/)r-r3r'rrr0rr(r rr1r2rresidualr*r+r1r1r2r?s"   zZambaMambaDecoderLayer.forward) NNNNNFFNNN)rDrErFrrr'r)rHrRr,rr@r-r?rIr1r1r/r2r.sJ     r.cseZdZdedejdeffdd Z        ddej d ej dBd e dBd ej dBd ej dBd e dBde dBde dBdej dBdeejeejejfdBffddZZS)ZambaHybridLayer shared_transflinearr/cs t||_||_||_dSr{)r&r'r5r6 mamba_decoder)r-r5r6r/r/r1r2r's  zZambaHybridLayer.__init__NFr3r'rrr0rr(r rr%c Csp|j|||||||| d} | d} |r| d} || } |j|| ||||| d} |r6| d| f| dd} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r'rrrr(r rrr)r2rrr(r rr4N)r5r6r7) r-r3r'rrr0rr(r r layer_outputsr2r*r1r1r2r?s4   zZambaHybridLayer.forward)NNNNNFFN)rDrErFr rrr.r'r)rHrrRr,rr@r-r?rIr1r1r/r2r4s>    r4csNeZdZUeed<dZdZddgZdZdZ dZ dZ e fdd ZZS) ZambaPreTrainedModelrxmodelTr r.rFcsD|jj}t|t|trtj|jd|d|jj d}t |j | ||jj |jj |jj}tt|jj|t|jjt|jjt|jjj|jjd}|tt|  }t|j|tjd|jdtjddddf}||jd }t|j!t|"|j|j#dt$|j%dSdS)Nr)r;stdr)minrrr5)&rxinitializer_ranger& _init_weightsrrinitnormal_rruniform_rrer.rkr)rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrhr9rKrfrrrLrones_r)r-rr; dt_init_stdrdtinv_dtrr/r1r2r>#s.    $"z"ZambaPreTrainedModel._init_weights)rDrErFr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr)no_gradr>rIr1r1r/r2r9s r9cseZdZdZdeffdd Ze          ddejdBdej dBdejdBd e dBd ej dBd e dBd e dBd e dBde dBdejdBde eBfddZZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rxcst||j|_|j|_t|j|j|j|_|j |_ g}d|_ t |j D]=\}}t ||d}|dkr_tj |jj|jjdd}|tt||||j dur^d|dd|di|_ q'||q't||_|j|_t|j|jd |_d|_|dS) N)rrUFrz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr!)r&r' pad_token_id padding_idx vocab_sizer Embeddingr. embed_tokensrc_tied_weights_keys enumerater.rrxrur4r  ModuleListlayersrr"r$final_layernormgradient_checkpointing post_init)r-rxr_layer_id layer_typer/r6r/r1r2r'Es,      zZambaModel.__init__N input_idsrr1r inputs_embedsr r(output_hidden_states return_dictrr%c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}|} t |} |r^|dur^t d| durmt j | jd| jd} |durv| d}t|j||| ||d}|rd nd}|rd nd}t|jD]+\}}|r|| f7}|| | ||||||| d }|d} |r|ddur||df7}q|| } |r|| f7}|r|jsd |_t| |r|nd||d }| r|S|S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrXr)rxrfrrrr1r1)rr(r rT)last_hidden_staterr3 attentions)rxr(rgr use_return_dictrrarrrr[r)rrrArTrr r]r_r`rdrto_tuple)r-rerr1rrfr r(rgrhrrr3r'r0all_hidden_statesall_self_attnsrlayerr8outputr1r1r2r?bs            zZambaModel.forward NNNNNNNNNN)rDrErFrrr'rr)rrHrRr-r,r@rr?rIr1r1r/r2rV<sJ    rVcseZdZddiZdeffdd Ze            ddejdBd ej dBd ejdBd e dBd ej dBd ejdBde dBde dBde dBde dBdejdBde ej BdeeBfddZ       dfdd ZZS)ZambaForCausalLMzlm_head.weightzmodel.embed_tokens.weightrxcs@t|t||_|j|_tj|j|jdd|_| dSr) r&r'rVr:rYrrr.lm_headrbrr/r1r2r's   zZambaForCausalLM.__init__Nrrerr1rrflabelsr r(rgrhrlogits_to_keepr%c  Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} |j|||||||| | | d }|d}t| tr>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rerr1rrfr r(rgrrhrrlosslogitsrr3rj)rxr(rgrkr:rrslicers loss_functionrYrrr3rj)r-rerr1rrfrtr r(rgrhrrurr+r3 slice_indicesrxrwrpr1r1r2r?s@( zZambaForCausalLM.forwardTFc sX|durt|j|jd|j|jd}|jj| d<tj|f|||||||d| } | S)Nr)r7rTru)rrrfrr1r is_first_iteration)rRrxrAr7rTnum_logits_to_keepr&prepare_inputs_for_generation) r-rerrrfrr1r r|r model_inputsr/r1r2r~+s&   z.ZambaForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNr)NNNNNTF)rDrErFr\rr'rr)rrHrRr-r,rr@rr?r~rIr1r1r/r2rrsf       Trra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcseZdZfddZe          ddejdBdejdBdejdBdedBdej dBd ejdBd e dBd e dBd e dBd e dBde e BfddZ ZS)ZambaForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSr) r&r' num_labelsrVr:rrr.scorerbrr/r1r2r']s   z'ZambaForSequenceClassification.__init__Nrerr1rrfrtr r(rgrhr%c  KsB| dur| n|jj} |j|||||||| | d } | d} || }|dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n1|durl||jjk|jt j }t j |jd|jt j d}|| d}n d}t |jjd|t j ||jd |f}d}|dur||j}|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rr1rrfr r(rgrhrrz=Cannot handle batch sizes > 1 if no padding token is defined.r5rSz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rX regressionsingle_label_classificationmulti_label_classificationrv)rxrkr:rrArWrr8rTr)int32rargmaxrrr0rD problem_typerr7longrrrrrrrrr3rj)r-rerr1rrfrtr r(rgrhrtransformer_outputsr3rxr_last_non_pad_token non_pad_mask token_indices pooled_logitsrwloss_fctrpr1r1r2r?fsx         "       z&ZambaForSequenceClassification.forwardrq)rDrErFr'rr)rrHr r-r,r@rr?rIr1r1r/r2rNsH      r)rrrrVr9)r)LrrCcollections.abcrtypingrr)rtorch.nnrrrr r? activationsr cache_utilsr generationr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1drr rr get_loggerrDrModuler"rHrrQrRrGrrrrr r.r4r9rVrrr__all__r1r1r1r2s                m F?EH$ j