o ei5@sddlZddlmZddlmZddlZddlmZm Z m Z e r%ddl Z e r,ddl Z ddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d d ed d ZGddded d ZeGdddeZdgZdS)N)Path)Any)auto_docstringis_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc@s&eZdZUdZeeefdBed<dS)CsmAudioKwargsa encoded_length_kwargs (`dict[str, Any]`, *optional*): Dictionary of keyword arguments used to compute the encoded audio sequence length. This includes parameters such as `kernel_sizes`, `strides`, `dilations`, and `use_causal_conv` that define the convolutional layers used in audio encoding. The encoded length is used to determine how many audio tokens to generate for each audio input in the text sequence. Nencoded_length_kwargs)__name__ __module__ __qualname____doc__dictstrr__annotations__rrd/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/csm/processing_csm.pyr$s rF)totalc@sJeZdZUeed<ddddgdgdgddd d d d d idZdS)CsmProcessorKwargs audio_kwargsTleftF)padding padding_sideadd_special_tokens)rrr$ rr$ rr$r)r$r$r$r)r$r$r$r$r$r$r%r$)r$r$r$r$r$r$r$r$r$r$r$r$r$r$r$) kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr common_kwargsN)rrrrr _defaultsrrrrr0s  rc seZdZ dfdd ZedddZdedeeBe eeBBde e fd d Z e  dd eeBe eBe eBdBdedBdedBdedBde e f ddZeddZZS) CsmProcessorNcsvt|dsd|_||j|_n|j|_|j|_t|ds(d|_||j|_n|j|_|j|_tj|||ddS)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>) chat_template)hasattrr8convert_tokens_to_idsaudio_token_idr9audio_eos_token_idsuper__init__)selffeature_extractor tokenizerr: __class__rrr@Gs  zCsmProcessor.__init__cCs|}|dus|dus|dus|dur|St|||D]U\}}}|d|d} ||} | d} | | } || | |d} t| d} | ||| }||}|rW| } |} n| |} || | }|||dd|d}q|S)a| Compute the length of the encoded audio sequence. Args: audio_length (int): The length of the audio sequence. kernel_sizes (list[int]): The kernel sizes for the convolutional layers. strides (list[int]): The strides for the convolutional layers. use_causal_conv (bool): Whether to use causal convolutions. Nr$r,)zipmathceil) audio_lengthr-r.r/r0 cur_length kernel_sizestridedilationeffective_kernel_size padding_total padding_right padding_leftn_frames ideal_length extra_paddingrrr_get_encoded_length]s&  z CsmProcessor._get_encoded_lengthaudio saving_pathkwargsc Kststdt|}t|ttfr|g}nt|ttfr&tdd|Ds*t dt |t |kr6t d|j t fi|}|d}|d}t ||D]\}}t|tjr^|}t|||qLdS)Nz/Please install `soundfile` to save audio files.css|] }t|ttfVqdSN) isinstancerr).0prrr sz*CsmProcessor.save_audio..zAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samerr1)r ImportErrorr rZrrlisttupleall ValueErrorlen _merge_kwargsrrFtorchTensorcpufloatnumpysfwrite) rArVrWrX output_kwargsrr1 audio_valuer\rrr save_audios*  zCsmProcessor.save_audioF?text output_labelsdepth_decoder_labels_ratioc sjtfdjji|}|d}|d}|dd} | dkr(tjjdt|t r1|g}nt|t t frAt dd |DsEtd fd d |D} d } |dur\t |}t|} t| d kr{| t| kr{|durptdtd| d| d|dur|difdd |D} | } g}|D]<}g}j|vr| d }j|}|||jdd}j|vsd|vr|d|d d}d|vs||q|}j|fi|}i}|||durs|ddgg}}d }| D]I} | d kr|td |tdgq|tjdd |||| Ddd|tdd |||| Djdd|| 7}qj|fi|}|dd||tdd |Dfdd |D}tj|d d|d<|r|d jk }|j!d }|d!krt"|dt#|d|}||}n|}t$|d jk|d j%kB|d d"}d#||ddd f|dddff<||d$<t&|| d%S)&a output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. tokenizer_init_kwargsr4rr2Nr3z% only supports `return_tensors='pt'`.css|]}t|tVqdSrY)rZrr[trrrr]z(CsmProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscsg|]}|jqSr)countr8rt)rArr z)CsmProcessor.__call__..rz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rcs$g|]}j|jdfiqS)rUshape)r[ audio_array)rrArrrxsz r$return_attention_maskr{cSs(g|]}t|tjr|n|qSr)rZrerfrgrir[elrrrrx s)axiscSsg|]}|jdqSrzr|rrrrrxs)dim padding_maskcss|]}|jdVqdS)r{Nrr[cut_idxsrrrr]rvcs.g|]}tjjj|d|jdfddqS)rr{)value)renn functionalpadr|r)max_lenrrrxs input_values_cutoffs input_idsroiilabels)data tensor_type)'rdrrC init_kwargsgetrbrErrZrr_r`rar rcsumpopcopyr8appendreplaceupdatenpzerosretensor concatenatecumsumrBmaxstackr=nonzeror|randpermintwherer>r )rArprVrqrrrXrlr4rr2n_audio_in_textn_audionum_audio_tokens_listnum_audio_tokens_list_copy expanded_textsample replace_strnum_audio_tokensexpanded_audio_tokenencodingrconcatenated_audioroffset audio_inputsaudio_frame_idxsn_audio_frames rand_idxsskip_frames_idxsrr)rrrAr__call__s                 &       $ zCsmProcessor.__call__cCs0|jj}|jj}dd|D}t||dgS)NcSsg|]}|dkr|qS)rr)r[namerrrrx>ryz2CsmProcessor.model_input_names..r)rCmodel_input_namesrBr_)rAtokenizer_input_namesfeature_extractor_input_namesrrrr7szCsmProcessor.model_input_namesrY)NNNN)NFro)rrrr@ staticmethodrUrrrr_rrrnrrrboolrhrpropertyr __classcell__rrrDrr7Es< % " r7)rGpathlibrtypingrrirutilsrrrre soundfilerj audio_utilsrr feature_extraction_utilsr processing_utilsr r r rtokenization_utils_baserrrrr7__all__rrrrs&     }