o
it @ s d dl mZ d dlmZmZ d dlZd dlZddlm Z m
Z
ddl m Z ddl
mZmZ e
eZe ed d d
G d d d eZdS )
)UserDict)AnyUnionN )add_end_docstringslogging ) ffmpeg_read)Pipelinebuild_pipeline_init_argsT)has_feature_extractor
has_tokenizerc s e Zd ZdZdZdZdZdZ fddZde e
j e e
ef dedeee
ef f fd d
Zd d ZdddZdd Zdd Z ZS )#ZeroShotAudioClassificationPipelinea
Zero shot audio classification pipeline using `ClapModel`. This pipeline predicts the class of an audio when you
provide an audio and a set of `candidate_labels`.
The default `hypothesis_template` is : `"This is a sound of {}."`. Make sure you update it for your usage.
Example:
```python
>>> from transformers import pipeline
>>> from datasets import load_dataset
>>> dataset = load_dataset("ashraq/esc50")
>>> audio = next(iter(dataset["train"]["audio"]))["array"]
>>> classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
>>> classifier(audio, candidate_labels=["Sound of a dog", "Sound of vacuum cleaner"])
[{'score': 0.9996, 'label': 'Sound of a dog'}, {'score': 0.0004, 'label': 'Sound of vacuum cleaner'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This audio
classification pipeline can currently be loaded from [`pipeline`] using the following task identifier:
`"zero-shot-audio-classification"`. See the list of available models on
[huggingface.co/models](https://huggingface.co/models?filter=zero-shot-audio-classification).
FTc s2 t jdi | | jdkrtd| j dd S )NptzThe z is only available in PyTorch. )super__init__ framework
ValueError __class__)selfkwargsr r i/home/ubuntu/.local/lib/python3.10/site-packages/transformers/pipelines/zero_shot_audio_classification.pyr D s
z,ZeroShotAudioClassificationPipeline.__init__audiosr returnc s t j|fi |S )a
Assign labels to the audio(s) passed as inputs.
Args:
audios (`str`, `list[str]`, `np.array` or `list[np.array]`):
The pipeline handles three types of inputs:
- A string containing a http link pointing to an audio
- A string containing a local path to an audio
- An audio loaded in numpy
candidate_labels (`list[str]`):
The candidate labels for this audio. They will be formatted using *hypothesis_template*.
hypothesis_template (`str`, *optional*, defaults to `"This is a sound of {}"`):
The format used in conjunction with *candidate_labels* to attempt the audio classification by
replacing the placeholder with the candidate_labels. Pass "{}" if *candidate_labels* are
already formatted.
Return:
A list of dictionaries containing one entry per proposed label. Each dictionary contains the
following keys:
- **label** (`str`) -- One of the suggested *candidate_labels*.
- **score** (`float`) -- The score attributed by the model to that label. It is a value between
0 and 1, computed as the `softmax` of `logits_per_audio`.
)r __call__)r r r r r r r K s z,ZeroShotAudioClassificationPipeline.__call__c K s6 i }d|v r |d |d< d|v r|d |d< |i i fS )Ncandidate_labelshypothesis_templater )r r preprocess_paramsr r r _sanitize_parametersd s
z8ZeroShotAudioClassificationPipeline._sanitize_parametersNThis is a sound of {}.c s t |tr/|ds|drt|j}nt|d }| }W d n1 s*w Y t |tr;t || j
j }t |t j
sEtdt|jdkrPtd| j
|g| j
j dd}| jdkrf|| j}||d < fd
d |D }| j|| jd d
}|g|d< |S )Nzhttp://zhttps://rbz"We expect a numpy ndarray as inputr zNWe expect a single channel audio input for ZeroShotAudioClassificationPipeliner )
sampling_ratereturn_tensorsr c s g | ]} |qS r )format).0xr r r
s zBZeroShotAudioClassificationPipeline.preprocess..T)r$ padding text_inputs)
isinstancestr
startswithrequestsgetcontentopenreadbytesr feature_extractorr# npndarray TypeErrorlenshaper r todtype tokenizer)r audior r finputs sequencesr+ r r( r
preprocessm s,
z.ZeroShotAudioClassificationPipeline.preprocessc C s\ | d}| d}t|d tr|d }n|d d }| jdi ||}||jd}|S )Nr r+ r )r logitsr )popr, r modellogits_per_audio)r model_inputsr r+ outputs
model_outputsr r r _forward s
z,ZeroShotAudioClassificationPipeline._forwardc C sb | d}|d d }| jdkr|jdd}| }ntddd tt||d d
d D }|S ) Nr rC r r )dimz `tf` framework not supported.c S s g | ] \}}||d qS ))scorelabelr )r& rL candidate_labelr r r r) s zCZeroShotAudioClassificationPipeline.postprocess..c S s
| d S )Nr r )r' r r r s
zAZeroShotAudioClassificationPipeline.postprocess..)key)rD r softmaxtolistr sortedzip)r rI r rC probsscoresresultr r r postprocess s
z/ZeroShotAudioClassificationPipeline.postprocess)Nr! )__name__
__module__ __qualname____doc___load_processor_load_image_processor_load_feature_extractor_load_tokenizerr r r6 r7 r4 r- dictr listr r rB rJ rX
__classcell__r r r r r s 4
r ) collectionsr typingr r numpyr6 r/ utilsr r audio_classificationr baser
r
get_loggerrY loggerr r r r r s