o i@sddlmZmZmZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZmZmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZm Z m!Z!m"Z"dd l#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2Gdddej3Z4Gdddej3Z5dej6de7dej6fddZ8 dLdej3dej6d ej6d!ej6d"eej6d#e9d$e9d%e*e,fd&d'Z:d(d)Z;dMd*d+ZGd0d1d1eZ?Gd2d3d3eZ@e-Gd4d5d5e(ZAGd6d7d7eAZBe-Gd8d9d9eAZC  dNd:eDe7e7fd;e9dd?ZGe-Gd@dAdAeAZHdBej6dCe7dDe7fdEdFZIe-dGdHGdIdJdJeAeZJgdKZKdS)O)CallableOptionalUnionN)OutputRecordercheck_model_inputs)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)MoonshineConfigc2eZdZfddZdejdejfddZZS)MoonshineEncoderMLPcsBt||_t||_t|j|j|_ t|j|j|_ dSN super__init__configr activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr( hidden_act __class__d/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/moonshine/modeling_moonshine.pyr'4s  zMoonshineEncoderMLP.__init__ hidden_statesreturncCs"||}||}||}|Sr$)r.r)r/)r1r7r5r5r6forward;s   zMoonshineEncoderMLP.forward__name__ __module__ __qualname__r'torchTensorr9 __classcell__r5r5r3r6r#3 r#cr")MoonshineDecoderMLPcsFt||_t||_t|j|jd|_ t|j|j|_ dS)Nr%r0r3r5r6r'Cs  zMoonshineDecoderMLP.__init__r7r8cCs8||}|jddd\}}|||}||}|S)NrCdim)r.chunkr)r/)r1r7gater5r5r6r9Js  zMoonshineDecoderMLP.forwardr:r5r5r3r6rBBrArBr7n_repr8cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)shapeexpandreshape)r7rIbatchnum_key_value_headsslenhead_dimr5r5r6 repeat_kvRs 0rQmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)NrCrrD)rFdtype)ptrainingr )rQnum_key_value_groupsr>matmul transposerJr* functionalsoftmaxfloat32tor\rYr^ contiguous) rSrTrUrVrWrXrYrZ key_states value_states attn_weights causal_mask attn_outputr5r5r6eager_attention_forward^s & rlcCs>|ddddf}|ddddf}tj| |fdddS) z*Rotates half the hidden dims of the input..rNrCr rDrEr[)r>stackflatten)xx1x2r5r5r6 rotate_halfxsrrc Cs||}||}|dd|jddfjddd}|dd|jddfjddd}|jd}|dd|f|d|df}}|dd|f|d|df} } ||t||} | |t| |} tj| |gdd} tj| | gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .NrDrCrE) unsqueezerJrepeat_interleaverrr>cat) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embedr5r5r6apply_rotary_pos_embs  $$ ""rcseZdZdZdededededef fdd Zed d d d  dde j de e e j e j fde e j d e e de e jde e j deede e j e e j e e e j ffddZZS)MoonshineAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr( layer_idx is_causalnum_attention_headsrNcst|||d||_||_t|d|j|j|_|j|j |_ |jd|_ |j |_ ||_ tj|j|j|j|jd|_tj|j|j |j|jd|_tj|j|j |j|jd|_tj|j|j|jdd|_|jjdur|jj}||j|d|}||j|_dSd|_dS)N)rrNrPg࿩biasFr r)r&r'updater(rgetattrr,rrPrNr_rXattention_dropoutrr*r+attention_biasq_projk_projv_projo_projpad_head_dim_to_multiple_ofhead_dim_padding)r1r(rrrrNtarget_multipletarget_head_dimr3r5r6r's0    zMoonshineAttention.__init__past_key_valuepast_key_values4.58new_nameversionNr7position_embeddingsrWcache_positionkey_value_statesrZr8cKsV|jdd\}} |||| |jj|jdd} |du} |dur9|j|j } | r6d|j|j <|j }n|j }|dur?|n|} | rV|rV| rV|j |j j }|j |j j}n7|| |d|jj|jdd}|| |d|jj|jdd}| r|dur||||j d|i\}}| s|\}}t| |||\} }|dur|||d}||||j |\}}t}|jjdkrt|jj}|jo|duo| dk}|jdkrtjj| d|jf} tjj|d|jf}tjj|d|jf}||| |||f|jsd n|j|j|d |\}}|jdkr|d d|j f}||| d }|!|}||fS) NrDr rCTr)ryrxreagerrrR)rYrXr.)"rJrviewr(rNrPra is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrrrrl_attn_implementationrrrr>r*rbpadr^rrXrLrfr)r1r7rrWrrrrZbszq_len query_statesis_cross_attentionrcurrent_statesrgrhrxry cache_kwargsattention_interfacerrkrir5r5r6r9sx "          zMoonshineAttention.forward)NNNNN)r;r<r=__doc__r!intboolr'rr>r?rtupler LongTensorrrr9r@r5r5r3r6rsH % rcsDeZdZUejed<ddeffdd Zee ddZ Z S) MoonshineRotaryEmbeddinginv_freqNr(cstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultrF) persistent)r&r'hasattr isinstancerdictrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr(r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r1r(devicerr3r5r6r'-s   z!MoonshineRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) NrrDr mpscpuF) device_typeenabledrCrEr\)rfloatrKrJrerrrstrr>autocastrarurxrryr\) r1rorzinv_freq_expandedposition_ids_expandedrfreqsembrxryr5r5r6r9>s0&z MoonshineRotaryEmbedding.forwardr$) r;r<r=r>r?__annotations__r!r'no_gradrr9r@r5r5r3r6r*s rcseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de ej de e ejejfdeedejfddZZS)MoonshineEncoderLayerr(rcs`t|j|_t||d|j|jd|_t||j|_ t j |jdd|_ t j |jdd|_ dS)NFr(rrrrNr)r&r'r,rencoder_num_attention_headsencoder_num_key_value_heads self_attnr#encoder_hidden_actmlpr* LayerNorminput_layernormpost_attention_layernormr1r(rr3r5r6r'Os zMoonshineEncoderLayer.__init__rrrrNFr7rWrz use_cacherrrZr8c Ks^|} ||}|jd|||||||d|\}} | |}|} ||}||}| |}|S)Nr7rWrzrrrrr5)rrrr) r1r7rWrzrrrrrZresidual_r5r5r6r9_s&     zMoonshineEncoderLayer.forward)NNNFNN)r;r<r=r!rr'rr>r?rrr rrrrr9r@r5r5r3r6rNs8   rc!seZdZddedeeffdd Zedddd          dd ej d eej d eej deej deej deej dee dee deej dee ej ej fdee ej ej fdeede ejee ejejfffddZZS)MoonshineDecoderLayerNr(rcst|j|_t||d|j|jd|_t||d|j|jd|_t||j |_ t j |jdd|_ t j |jdd|_t j |jdd|_dS)NTrFr)r&r'r,rdecoder_num_attention_headsdecoder_num_key_value_headsr encoder_attnrBdecoder_hidden_actrr*rrrfinal_layernormrr3r5r6r's( zMoonshineDecoderLayer.__init__rrrrFr7rWencoder_hidden_statesencoder_attention_maskrzencoder_position_idsrrrencoder_position_embeddingsrZr8c  Ks|} ||}|jd|||||| | d| \}}| |}|dur8|} ||}|j|||||d\}}| |}|} ||}||}| |}|S)Nr)r7rrWrrr5)rrrrrr)r1r7rWrrrzrrrrrrrZrrr5r5r6r9s<       zMoonshineDecoderLayer.forwardr$) NNNNNNFNNN)r;r<r=r!rrr'rr>r?rr rrrr FloatTensorr9r@r5r5r3r6rsP      rc@sFeZdZUeed<dZdZdZddgZdZ dZ dZ de j fdd Zd S) MoonshinePreTrainedModelr(model input_valuesTrr input_lengthscCs@t|ddd}t|ddd}t|ddd}|S)zH Computes the output length of the convolutional layers @r rrC)r)r1routput_conv1_lengthoutput_conv2_lengthoutput_conv3_lengthr5r5r6 _get_feat_extract_output_lengthssz9MoonshinePreTrainedModel._get_feat_extract_output_lengthsN)r;r<r=r!rbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_can_compile_fullgraphr>rrr5r5r5r6rs rc seZdZdZdZeedZdeffdd Z de j fdd Z d e j fd d Z e ddejdeejdeedefddZZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig r) attentionsr7r(cst|_j}tjd|dddd|_tj|d|ddd |_tjd||ddd |_tj d|d d |_ t d |_ t fd dtjD|_tj|dd|_d|_|dS)Nr rrF) kernel_sizestriderrCrr)rrgh㈵>) num_groups num_channelsepsr(cg|]}t|qSr5)r.0idxr r5r6 z-MoonshineEncoder.__init__..r)r&r'r(r,r*Conv1dconv1conv2conv3 GroupNorm groupnormr rotary_emb ModuleListrangeencoder_num_hidden_layersrr layer_normgradient_checkpointing post_init)r1r( embed_dimr3r r6r's   zMoonshineEncoder.__init__r8cC|jSr$rr1r5r5r6get_input_embeddingsz%MoonshineEncoder.get_input_embeddingsrVcC ||_dSr$rr1rVr5r5r6set_input_embeddings z%MoonshineEncoder.set_input_embeddingsNrWrZc Ks<|d}tj||}||}tj||}tj||}| ddd}|durm| |j d}d}|ddd|fdd|f}|j j dkrZ|d krW|nd}n|j j d krgt||j}nt||j}tjd|j d|jd d}|||}|jD]} | |f|||d |}q||}t|d S)a. Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) r rrCNrD.flash_attention_2rRsdpar)rWrzr)last_hidden_state)rsr*rbtanhrrgelurrpermuterrJr(ranyrr\rr>arangerrrrr) r1rrWrZr7mask_lendownsample_striderzr encoder_layerr5r5r6r9 s>        zMoonshineEncoder.forwardr$)r;r<r=rrrr_can_record_outputsr!r'r*Moduler!r%rr>rrr?rrrr9r@r5r5r3r6rs(rcseZdZdZeedddeeeddddZdeffdd Z e dde e j d e e jd e e j d e ede e jde ede e j de e jde e jdeedeeeffddZZS)MoonshineDecoder input_idsr r)index layer_namer)rr7cross_attentionsr(cstj|_j|_tjj|j|_t fddt j D|_ tj jdd|_td|_d|_|dS)Ncr r5)rr r r5r6rWrz-MoonshineDecoder.__init__..Frr )r&r' pad_token_id padding_idx vocab_sizer* Embeddingr, embed_tokensrrdecoder_num_hidden_layersrrnormrrrrr1r(r3r r6r'Ps   zMoonshineDecoder.__init__NrWrzr inputs_embedsrrrrrZr8c  Ks|du|duAr td|dur||}|r(|dur(tt|jdt|jd}|durD|dur4|nd} tj| | |jd|j d}|durM| d}t |j|||||d} |} | | |}| dur|jd}d }| d dd|fd d|f} |jj d kr| d kr| nd} n|jj d krt| | j| jd} n t| | j| jd} |jD]}|| | |f| |||||d| } q|| } t| |r|dSddS)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedsr rr r*)r( input_embedsrWrrrzr[r'.r(rRr))rrzrrrr)r+r) ValueErrorr?r r r(get_seq_lengthr>r0rJrrsr rrr/rr\rrrAr)r1r7rWrzrrCrrrrrZpast_seen_tokensrjr7rr1r2 decoder_layerr5r5r6r9`st          zMoonshineDecoder.forward) NNNNNNNNN)r;r<r=rrrrr4r!r'rrr>rr?r rrrrrrrr9r@r5r5r3r6r6GsP      r6rJ mask_prob mask_length min_maskscs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. r z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedr r)rmax) input_lengthnum_masked_spanepsilonrJrIrKsequence_lengthr5r6compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNrDcsg|]}qSr5r5)r r)rRr5r6rsz)_compute_mask_indices..rrF)replace)rEnprandomranditemdetachsumtolistrzerosrchoicer0len concatenateonesint32appendarray broadcast_torLrMput_along_axis)rJrIrJrWrK batch_sizerSr spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrNrOspec_aug_mask_idxdummy_mask_idxoffsetsr5rPr6_compute_mask_indicess\    rmcseZdZdeffdd ZddZddZdd Zd d Z dd e j de e j fddZ ee dde e j de e j de e j de e j de eee j de eeee j fde ee j de ee j de ede e j deedefddZZS) MoonshineModelr(cs,t|t||_t||_|dSr$)r&r'rencoderr6decoderrrBr3r5r6r'4s    zMoonshineModel.__init__cCs|jjSr$rpr?r r5r5r6r!<sz#MoonshineModel.get_input_embeddingscCs ||j_dSr$rqr$r5r5r6r%?s z#MoonshineModel.set_input_embeddingscCrr$)ror r5r5r6 get_encoderBr"zMoonshineModel.get_encodercCs|jdS)z Calling this function will disable the gradient computation for the Moonshine encoder so that its parameters will not be updated during training. N)ro_freeze_parametersr r5r5r6freeze_encoderEszMoonshineModel.freeze_encoderNinput_featuresrWcCst|jdds |S|\}}}|jjdkrE|jrEt||f|jj|jj||jjd}tj ||j tj d}|dddf d|d}d||<|jj dkrl|jrlt||f|jj |jj|jjd}tj ||j tj d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTr)rIrJrWrK)rr\NrD)rIrJrK)rr(sizemask_time_probr^rmmask_time_lengthmask_time_min_masksr>tensorrrrKmask_feature_probmask_feature_lengthmask_feature_min_masks)r1rurWrfr,rRmask_time_indicesmask_feature_indicesr5r5r6_mask_input_featuresLs0 z#MoonshineModel._mask_input_featuresrdecoder_input_idsdecoder_attention_maskencoder_outputsrdecoder_inputs_embedsdecoder_position_idsrrrZr8c Ksl|dur|j|fd|i| }|jd||||j|||| | d | } t| j| j| j| j| j|j|j|jdS)a  input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ``` NrW) r7rWrrrrCrzrr)r+rdecoder_hidden_statesdecoder_attentionsr:encoder_last_hidden_staterencoder_attentionsr5)rorpr+rrr7rr:) r1rrWrrrrrrrrrZdecoder_outputsr5r5r6r9ws2.  zMoonshineModel.forwardr$) NNNNNNNNNN)r;r<r=r!r'r!r%rrrtr>rrrrrrrrr rrrrr9r@r5r5r3r6rn2sb  +      rnr7r;decoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. NrDr rz1self.model.config.pad_token_id has to be defined.i) new_zerosrJclonerE masked_fill_)r7r;rshifted_input_idsr5r5r6shift_tokens_rights (rzj The Moonshine Model with a language modeling head. Can be used for automatic speech recognition. ) custom_introcseZdZdgZdeffdd ZddZddZd d Zd d Z d e j fddZ e e           ddeejdeejdeejdeejdeeeejdeeeeejfdeeejdeeejdeedeejdeejdeed efddZZS) !MoonshineForConditionalGenerationzproj_out.weightr(cs8t|t||_tj|j|jdd|_| dS)NFr) r&r'rnrr*r+r,r=proj_outrrBr3r5r6r's   z*MoonshineForConditionalGeneration.__init__cC |jSr$)rrrr r5r5r6rrr&z-MoonshineForConditionalGeneration.get_encodercCrr$)r get_decoderr r5r5r6rr&z-MoonshineForConditionalGeneration.get_decodercCrr$rr r5r5r6get_output_embeddingsr"z7MoonshineForConditionalGeneration.get_output_embeddingscCr#r$r)r1new_embeddingsr5r5r6set_output_embeddingsr&z7MoonshineForConditionalGeneration.set_output_embeddingsr8cCrr$)rr!r r5r5r6r!r&z6MoonshineForConditionalGeneration.get_input_embeddingsNrrWrrrrrrrrlabelsrZc  Ks| dur|dur|durt| |jj|jj}|j|f|||||||| | d | } || j}d}| dur?|j|| |jjd}t ||| j | j | j | j | j| j| jd S)a0 input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N) rWrrrrrrrr)logitsrr=) lossrrrrr:rrr)rr(r;rrrr+ loss_functionr=rrrrr:rrr)r1rrWrrrrrrrrrrZoutputsrrr5r5r6r9sF3  z)MoonshineForConditionalGeneration.forward) NNNNNNNNNNN)r;r<r=_tied_weights_keysr!r'rrrrrr*r5r!rrrr>rrrrr rrrrr9r@r5r5r3r6rs`      r)rnrr)rR)Nr )Nr)LtypingrrrnumpyrUr>torch.nnr*transformers.utils.genericrr activationsr cache_utilsr r r generationr masking_utilsr modeling_attn_mask_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationrconfiguration_moonshiner!r5r#rBr?rrQrrlrrrrrrrrrr6rrndarrayrmrnrr__all__r5r5r5r6s           *$4Kbw  w s