o ۷i@@sddlZddlmZddlmZmZmZddlZddl m Z m Z e r'ddl Z e r.ddl ZddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d d ed d ZGddded d ZGdddeZdgZdS)N)Path)AnyOptionalUnion)is_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc@s"eZdZUeeeefed<dS)CsmAudioKwargsencoded_length_kwargsN)__name__ __module__ __qualname__rdictstrr__annotations__rr\/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/csm/processing_csm.pyr%s rF)totalc@sJeZdZUeed<ddddgdgdgddd d d d d idZdS)CsmProcessorKwargs audio_kwargsTleftF)padding padding_sideadd_special_tokens)rrr$ rr$ rr$r)r$r$r$r)r$r$r$r$r$r$r%r$)r$r$r$r$r$r$r$r$r$r$r$r$r$r$r$) kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr common_kwargsN)rrrrr _defaultsrrrrr)s  rc seZdZdZddgZdZdZ dfdd Zedd d Z d e d e e e ee e e ffd eefddZ   ddee eeeeeefd ee deedeed eef ddZeddZZS) CsmProcessora  Constructs a Csm processor which wraps [`EncodecFeatureExtractor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the audio feature extraction and tokenizer functionalities. See the [`~CsmProcessor.__call__`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import CsmProcessor from datasets import load_dataset ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") audio = ds[0]["audio"]["array"] processor = CsmProcessor.from_pretrained("sesame/csm-1b") processor( text=["<|begin_of_text|>[0]What are you working on?<|end_of_text|><|AUDIO|><|audio_eos|><|begin_of_text|>[1]I'm figuring out my budget.<|end_of_text|>"], audio=audio, text_kwargs = {"padding": False}, audio_kwargs = {"sampling_rate": 16000}, common_kwargs = {"return_tensors": "pt"}, ) # this should error out because EncodecFeatureExtractor expects a 24kHz audio :) ``` Args: feature_extractor ([`EncodecFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. feature_extractor tokenizerEncodecFeatureExtractorPreTrainedTokenizerFastNcsvt|dsd|_||j|_n|j|_|j|_t|ds(d|_||j|_n|j|_|j|_tj|||ddS)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>) chat_template)hasattrr<convert_tokens_to_idsaudio_token_idr=audio_eos_token_idsuper__init__)selfr8r9r> __class__rrrDfs  zCsmProcessor.__init__cCs|}|dus|dus|dus|dur|St|||D]U\}}}|d|d} ||} | d} | | } || | |d} t| d} | ||| }||}|rW| } |} n| |} || | }|||dd|d}q|S)a| Compute the length of the encoded audio sequence. Args: audio_length (int): The length of the audio sequence. kernel_sizes (list[int]): The kernel sizes for the convolutional layers. strides (list[int]): The strides for the convolutional layers. use_causal_conv (bool): Whether to use causal convolutions. Nr$r,)zipmathceil) audio_lengthr-r.r/r0 cur_length kernel_sizestridedilationeffective_kernel_size padding_total padding_right padding_leftn_frames ideal_length extra_paddingrrr_get_encoded_length|s&  z CsmProcessor._get_encoded_lengthaudio saving_pathkwargsc Kststdt|}t|ttfr|g}nt|ttfr&tdd|Ds*t dt |t |kr6t d|j t fi|}|d}|d}t ||D]\}}t|tjr^|}t|||qLdS)Nz/Please install `soundfile` to save audio files.css|] }t|ttfVqdSN) isinstancerr).0prrr sz*CsmProcessor.save_audio..zAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samerr1)r ImportErrorr r\rrlisttupleall ValueErrorlen _merge_kwargsrrHtorchTensorcpufloatnumpysfwrite) rErXrYrZ output_kwargsrr1 audio_valuer^rrr save_audios*  zCsmProcessor.save_audioF?text output_labelsdepth_decoder_labels_ratioc sjtfdjji|}|d}|d}|d} | dd} | dkr,tjjdt|t r5|g}nt|t t frEt d d |DsItd fd d |D} d} |dur`t |}t|} t| dkr| t| kr|durttdtd| d| d|dur|difdd |D} | }g}|D]<}g}j|vr|d}j|}|||jdd}j|vsd|vr|d|dd}d|vs||q|}j|fi|}i}|||durw|ddgg}}d}| D]I} | dkr|td|tdgq|tjdd |||| Ddd|tdd |||| Djdd|| 7}qj|fi|}|dd||tdd |Dfdd |D}tj|dd|d <|r|d!jk}|j d}|d"krt!|dt"|d|}||}n|}t#|d!jk|d!j$kB|d!d#}d$||dddf|dddff<||d%<t%|| d&S)'a Main method to prepare text(s) and audio to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the audio, this method forwards the `audio` arguments to EncodecFeatureExtractor's [`~EncodecFeatureExtractor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. tokenizer_init_kwargsr4rr5r2Nr3z% only supports `return_tensors='pt'`.css|]}t|tVqdSr[)r\rr]trrrr_z(CsmProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscsg|]}|jqSr)countr<rv)rErr z)CsmProcessor.__call__..rz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rcs$g|]}j|jdfiqS)rWshape)r] audio_array)rrErrrzsz r$return_attention_maskr}cSs(g|]}t|tjr|n|qSr)r\rgrhrirkr]elrrrrz=s)axiscSsg|]}|jdqSr|r~rrrrrzEs)dim padding_maskcss|]}|jdVqdS)r}Nrr]cut_idxsrrrr_Nrxcs.g|]}tjjj|d|jdfddqS)rr})value)rgnn functionalpadr~r)max_lenrrrzOs input_values_cutoffs input_idsrqiilabels)data tensor_type)&rfrr9 init_kwargspoprdrGrr\rrarbrcr resumcopyr<appendreplaceupdatenpzerosrgtensor concatenatecumsumr8maxstackrAnonzeror~randpermintwhererBr )rErrrXrsrtrZrnr4rr5r2n_audio_in_textn_audionum_audio_tokens_listnum_audio_tokens_list_copy expanded_textsample replace_strnum_audio_tokensexpanded_audio_tokenencodingrconcatenated_audioroffset audio_inputsaudio_frame_idxsn_audio_frames rand_idxsskip_frames_idxsrr)rrrEr__call__s/                 &       $ zCsmProcessor.__call__cCs0|jj}|jj}dd|D}t||dgS)NcSsg|]}|dkr|qS)rr)r]namerrrrzqr{z2CsmProcessor.model_input_names..r)r9model_input_namesr8ra)rEtokenizer_input_namesfeature_extractor_input_namesrrrrjszCsmProcessor.model_input_namesr[)NNNN)NFrq)rrr__doc__ attributesfeature_extractor_classtokenizer_classrD staticmethodrWr rrrrarrrprrrboolrjrpropertyr __classcell__rrrFrr7>sB# % % 'r7) rIpathlibrtypingrrrrkrutilsrrrg soundfilerl audio_utilsr r feature_extraction_utilsr processing_utilsr r rrtokenization_utils_baserrrrr7__all__rrrrs&   9