o ei@sddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZmZddl m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0m1Z1ddl2m3Z3ee*ddGdddeZ4Gdddej5Z6Gdddej5Z7Gdddej5Z8d ej9d!e:d"ej9fd#d$Z; %dKd&ej5d'ej9d(ej9d)ej9d*ej9dBd+edLd2d3Z?ee?Gd4d5d5ej5Z@Gd6d7d7eZAGd8d9d9eZBe*Gd:d;d;e%ZCGdd?d?eCZEe*Gd@dAdAeCZFdBej9dCe:dDe:fdEdFZGe*dGdGdHdIdIeCeZHgdJZIdS)M)Callable) dataclass)OptionalN)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)use_kernelized_func)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs)MoonshineConfigz Extends [~modeling_outputs.BaseModelOutput] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc@s eZdZUdZejdBed<dS)MoonshineEncoderModelOutputNattention_mask)__name__ __module__ __qualname__r%torchTensor__annotations__r,r,n/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/moonshine/modeling_moonshine.pyr$3s r$c2eZdZfddZdejdejfddZZS)MoonshineEncoderMLPcsBt||_t||_t|j|j|_ t|j|j|_ dSN super__init__configr activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr4 hidden_act __class__r,r-r3>s  zMoonshineEncoderMLP.__init__ hidden_statesreturncCs"||}||}||}|Sr0)r:r5r;)r=rAr,r,r-forwardEs   zMoonshineEncoderMLP.forwardr&r'r(r3r)r*rC __classcell__r,r,r?r-r/= r/cr.)MoonshineDecoderMLPcsFt||_t||_t|j|jd|_ t|j|j|_ dS)Nr1r<r?r,r-r3Ms  zMoonshineDecoderMLP.__init__rArBcCs8||}|jddd\}}|||}||}|S)NrHdim)r:chunkr5r;)r=rAgater,r,r-rCTs  zMoonshineDecoderMLP.forwardrDr,r,r?r-rGLrFrGc s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)MoonshineRotaryEmbeddinginv_freqNr4cst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrOF) persistentoriginal_inv_freq) r2r3max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr4rope_parametersrPcompute_default_rope_parametersrattention_scalingregister_bufferclone)r=r4device rope_init_fnrOr?r,r-r3_s   z!MoonshineRotaryEmbedding.__init__r\z torch.deviceseq_lenrBz torch.Tensorc Cst|jd}|jdd}t|ddp|j|j}t||}d}d|tjd|dtjdj |tj d |}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrrHdtype)r\rc) rWgetgetattrr8num_attention_headsintr)arangeint64tofloat) r4r\r^baser`rarKattention_factorrOr,r,r-rXos  &z8MoonshineRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) NrrIr!mpscpuF) device_typeenabledrHrJrb)rOrkexpandshaperjr\ isinstancetypestrr transposer)catcosrYsinrc) r=x position_idsinv_freq_expandedposition_ids_expandedrpfreqsembryrzr,r,r-rCs0&z MoonshineRotaryEmbedding.forwardr0)NNN)r&r'r(r)r*r+r"r3 staticmethodrrgtuplerkrXno_gradrrCrEr,r,r?r-rN\s&   rNrAn_reprBcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rsrrreshape)rArbatchnum_key_value_headsslenrar,r,r- repeat_kvs 0rmodulequerykeyvaluer%scalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)NrHrrI)rKrc)ptrainingr!)rnum_key_value_groupsr)matmulrwr6 functionalsoftmaxfloat32rjrcrr contiguous) rrrrr%rrr key_states value_states attn_weights attn_outputr,r,r-eager_attention_forwards  rcCs>|ddddf}|ddddf}tj| |fdddS) z*Rotates half the hidden dims of the input..rNrHr!rIrJ)r)stackflatten)r{x1x2r,r,r- rotate_halfsrc Cs||}||}|dd|jddfjddd}|dd|jddfjddd}|jd}|dd|f|d|df}}|dd|f|d|df}} ||t||} ||t||} tj| |gdd} tj| | gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .NrIrHrJ) unsqueezersrepeat_interleaverr)rx) qkryrz unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embedr,r,r-apply_rotary_pos_embs  $$ ""rcseZdZdZdededededef fdd Z dd ej d e ej ej fd Bd ej d Bd e d Bdej d Bdej d Bde ede ej ej d Be ej d BffddZZS)MoonshineAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr4 layer_idx is_causalrfrcst|||d||_||_t|d|j|j|_|j|j |_ |jd|_ |j |_ ||_ tj|j|j|j|jd|_tj|j|j |j|jd|_tj|j|j |j|jd|_tj|j|j|jdd|_|jjdur|jj}||j|d|}||j|_dSd|_dS)N)rfrrag࿩biasFr!r)r2r3updater4rrer8rfrarrrattention_dropoutrr6r7attention_biasq_projk_projv_projo_projpad_head_dim_to_multiple_ofhead_dim_padding)r=r4rrrfrtarget_multipletarget_head_dimr?r,r-r3s0    zMoonshineAttention.__init__NrAposition_embeddingsr%past_key_valuescache_positionkey_value_statesrrBcKsJ|jdd\}} |||| |jj|jdd} |du} |dur9|j|j } | r6d|j|j <|j }n|j }|dur?|n|} | rV|rV| rV|j |j j }|j |j j}n7|| |d|jj|jdd}|| |d|jj|jdd}| r|dur||||j d|i\}}| s|\}}t| |||\} }|dur|||d}||||j |\}}t|jjt}|jo|duo| dk}|jdkrtjj| d|jf} tjj|d|jf}tjj|d|jf}||| |||f|jsdn|j|j|d |\}}|jdkr|d d|j f}| || d!}|"|}||fS) NrIr!rHTr)rzryrrr)rrr.)#rsrviewr4rrarw is_updatedrdrcross_attention_cacheself_attention_cachelayerskeysvaluesrrrrr get_interface_attn_implementationrrrr)r6rpadrrrrrr)r=rArr%rrrrbszq_len query_statesis_cross_attentionrcurrent_statesrrryrz cache_kwargsattention_interfacerrrr,r,r-rCsx "        zMoonshineAttention.forward)NNNNN)r&r'r(__doc__r"rgboolr3r)r*rr LongTensorrrrCrEr,r,r?r-rsF( rcseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e dejfddZZS)MoonshineEncoderLayerr4rcs`t|j|_t||d|j|jd|_t||j|_ t j |jdd|_ t j |jdd|_ dS)NFr4rrrfrr)r2r3r8rencoder_num_attention_headsencoder_num_key_value_heads self_attnr/encoder_hidden_actmlpr6 LayerNorminput_layernormpost_attention_layernormr=r4rr?r,r-r3us zMoonshineEncoderLayer.__init__NFrAr%r|r use_cacherrrrBc Ks^|} ||}|jd|||||||d|\}} | |}|} ||}||}| |}|S)NrAr%r|rrrrr,)rrrr) r=rAr%r|rrrrrresidual_r,r,r-rCs&     zMoonshineEncoderLayer.forward)NNNFNN)r&r'r(r"rgr3r)r*rrrrrrrCrEr,r,r?r-rts6  rcseZdZddededBffdd Z          ddejdejdBd ejdBd ejdBd ejdBd ejdBd e dBde dBdejdBde ejejfdBde ejejfdBde e de eje ejejfdBffddZZS)MoonshineDecoderLayerNr4rcst|j|_t||d|j|jd|_t||d|j|jd|_t||j |_ t j |jdd|_ t j |jdd|_t j |jdd|_dS)NTrFr)r2r3r8rrfrr encoder_attnrGr>rr6rrrfinal_layernormrr?r,r-r3s( zMoonshineDecoderLayer.__init__FrAr%encoder_hidden_statesencoder_attention_maskr|encoder_position_idsrrrrencoder_position_embeddingsrrBc  Ks|} ||}|jd|||||| | d| \}}| |}|dur8|} ||}|j|||||d\}}| |}|} ||}||}| |}|S)Nr)rArr%rrr,)rrrrrr)r=rAr%rrr|rrrrrrrrrr,r,r-rCs<       zMoonshineDecoderLayer.forwardr0) NNNNNNFNNN)r&r'r(r"rgr3r)r*rrrrrr FloatTensorrCrEr,r,r?r-rsN     rc@sJeZdZUeed<dZdZdZdZddgZ dZ dZ dZ de jfd d Zd S) MoonshinePreTrainedModelr4model input_valuesaudioTrr input_lengthscCs@t|ddd}t|ddd}t|ddd}|S)zH Computes the output length of the convolutional layers @r!rrH)rg)r=routput_conv1_lengthoutput_conv2_lengthoutput_conv3_lengthr,r,r- _get_feat_extract_output_lengthssz9MoonshinePreTrainedModel._get_feat_extract_output_lengthsN)r&r'r(r"r+base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_can_compile_fullgraphr)rrr,r,r,r-rs rc seZdZdZdZeedZdeffdd Z de j fdd Z d e j fd d Z ee ddejdejd BdeedeeBfddZZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig r) attentionsrAr4cst|_j}tjd|dddd|_tj|d|ddd |_tjd||ddd |_tj d|d d |_ t fd d t j D|_tj|dd|_td|_d|_|dS)Nr!rrF) kernel_sizestriderrHrr)rrgh㈵>) num_groups num_channelsepscg|]}t|qSr,)r.0idxr4r,r- #z-MoonshineEncoder.__init__..rr)r2r3r4r8r6Conv1dconv1conv2conv3 GroupNorm groupnorm ModuleListrangeencoder_num_hidden_layersrr layer_normrN rotary_embgradient_checkpointing post_init)r=r4 embed_dimr?rr-r3s   zMoonshineEncoder.__init__rBcC|jSr0rr=r,r,r-get_input_embeddings*z%MoonshineEncoder.get_input_embeddingsrcC ||_dSr0r"r=rr,r,r-set_input_embeddings- z%MoonshineEncoder.set_input_embeddingsNr%rc Ks"|d}tj||}||}tj||}tj||}| ddd}|durK| |j d}d}|ddd|fdd|f}|}t |j |||d}tjd|j d|jd d}|j||d } |jD]} | |f||| d |}qm||}t||dur|d Sdd S) a. Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) r!rrHNrIi.r4 inputs_embedsr%rr\r|)r%r|r)last_hidden_stater%)rr6rtanhrrgelurrpermuterrsr r4r)rhr\rrrr$rg) r=rr%rrAmask_lendownsample_strideoutput_attention_maskr|r encoder_layerr,r,r-rC0sH    zMoonshineEncoder.forwardr0)r&r'r(rrrr_can_record_outputsr"r3r6Moduler$r(rr r)rr*rrrrrCrEr,r,r?r-r s*rcseZdZdZeedddeeeddddZdeffdd Z e e dde j d Bd e jd Bd e j d Bd ed Bde jd Bded Bde j d Bde jd Bde jd BdeedeeBfddZZS)MoonshineDecoder input_idsr!r)index layer_namer)rrAcross_attentionsr4cstj|_j|_tjj|j|_t fddt j D|_ tj jdd|_td|_d|_|dS)Ncr r,)rr rr,r-rrz-MoonshineDecoder.__init__..Frr)r2r3 pad_token_id padding_idx vocab_sizer6 Embeddingr8 embed_tokensrrnum_hidden_layersrrnormrNrrrr=r4r?rr-r3ys    zMoonshineDecoder.__init__Nr%r|rr+rrrrrrBc  Ks$|du|duAr td|dur||}|r(|dur(tt|jdt|jd}|durD|dur4|nd} tj| | |jd|j d}|durM| d}t |j|||||d} t |j|| |d} |} |j | |d }|jD]}|| | |f| |||||d | } qm|| } t| |r|d Sdd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedsrrr!r,)r4r+r%rrr|r*r-)rr|rrrr)r.r) ValueErrorrAr rr4get_seq_lengthr)rhrsr\rr r rrrCr)r=r9r%r|rr+rrrrrpast_seen_tokens causal_maskrAr decoder_layerr,r,r-rCsf      zMoonshineDecoder.forward) NNNNNNNNN)r&r'r(rrrrr6r"r3rr r)rr*rrrrrrrrCrEr,r,r?r-r8psR      r8cseZdZdeffdd ZddZddZdd Zd d Ze e dd e j d Bde j d Bde j d Bde j d Bdeee j d Bded Bdee j d Bdee j d Bded Bde j d BdeedefddZZS)MoonshineModelr4cs,t|t||_t||_|dSr0)r2r3rencoderr8decoderrrDr?r,r-r3s    zMoonshineModel.__init__cCs|jjSr0rLrAr#r,r,r-r$sz#MoonshineModel.get_input_embeddingscCs ||j_dSr0rMr'r,r,r-r(s z#MoonshineModel.set_input_embeddingscCs|jdS)z Calling this function will disable the gradient computation for the Moonshine encoder so that its parameters will not be updated during training. N)rK_freeze_parametersr#r,r,r-freeze_encoderszMoonshineModel.freeze_encodercCstd)z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). zNot needed for Moonshine)AttributeErrorr#r,r,r-_mask_input_featuressz#MoonshineModel._mask_input_featuresNrr%decoder_input_idsdecoder_attention_maskencoder_outputsrdecoder_inputs_embedsdecoder_position_idsrrrrBc Ksn|dur|j|fd|i| }|jd|||j|j|||| | d | } t| j| j| j| j| j|j|j|jdS)a  input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ``` Nr%) r9r%rrrr+r|rr)r.rdecoder_hidden_statesdecoder_attentionsr<encoder_last_hidden_staterencoder_attentionsr,) rKrLr.r%rrrArr<) r=rr%rRrSrTrrUrVrrrdecoder_outputsr,r,r-rCs2.  zMoonshineModel.forward) NNNNNNNNNN)r&r'r(r"r3r$r(rOrQrrr)rrrr rrrrrCrEr,r,r?r-rJsV      rJr9r=decoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. NrIr!rz1self.model.config.pad_token_id has to be defined.i) new_zerosrsr[rE masked_fill_)r9r=r\shifted_input_idsr,r,r-shift_tokens_rightBs (r`zj The Moonshine Model with a language modeling head. Can be used for automatic speech recognition. cseZdZddiZdeffdd ZddZdd Zd ej fd d Z e e dde jd Bde jd Bde jd Bde jd Bdeee jd Bded Bdee jd Bdee jd Bded Bde jd Bde jd Bdeed efddZZS)!MoonshineForConditionalGenerationzproj_out.weightz!model.decoder.embed_tokens.weightr4cs8t|t||_tj|j|jdd|_| dS)NFr) r2r3rJrr6r7r8r?proj_outrrDr?r,r-r3Zs   z*MoonshineForConditionalGeneration.__init__cCr!r0rbr#r,r,r-get_output_embeddingsbr%z7MoonshineForConditionalGeneration.get_output_embeddingscCr&r0rc)r=new_embeddingsr,r,r-set_output_embeddingser)z7MoonshineForConditionalGeneration.set_output_embeddingsrBcCs |jSr0)rr$r#r,r,r-r$hr)z6MoonshineForConditionalGeneration.get_input_embeddingsNrr%rRrSrTrrUrVrrlabelsrc  Ks| dur|dur|durt| |jj|jj}|j|f|||||||| | d | } || j}d}| dur?|j|| |jjd}t ||| j | j | j | j | j| j| jd S)a0 input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N) r%rRrTrSrrUrVrr)logitsrgr?) lossrhrrWrXr<rYrrZ)r`r4r=r\rrbr. loss_functionr?rrrWrXr<rYrrZ)r=rr%rRrSrTrrUrVrrrgroutputsrhrir,r,r-rCksF3  z)MoonshineForConditionalGeneration.forward) NNNNNNNNNNN)r&r'r(_tied_weights_keysr"r3rdrfr6r7r$rrr)rrrr rrrrrCrEr,r,r?r-raRs\      ra)rJrra)r)r!)Jcollections.abcr dataclassesrtypingrr)torch.nnr6 activationsr cache_utilsrrr generationr integrationsr masking_utilsr r modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrr configuration_moonshiner"r$r7r/rGrNr*rgrrkrrrrrrrrr8rJr`ra__all__r,r,r,r-s           C  (3Jfig m