o ϯi"@sdZddlZddlZddlZddlZddlmZddlmZddlm Z m Z ddl m Z ddl Z ddlmZGd d d ejjZdS) z Contrastive Language-Audio Pretraining Model from LAION -------------------------------------------------------- Paper: https://arxiv.org/abs/2211.06687 Authors (equal contributions): Ke Chen, Yusong Wu, Tianyu Zhang, Yuchen Hui Support: LAION N) create_model)get_audio_features)int16_to_float32float32_to_int16)RobertaTokenizer)load_state_dictcsPeZdZddfdd Zdd Zdd d ZdddZdddZdddZZ S) CLAP_ModuleFN HTSAT-tinyrobertareturnc stt||durtjrdnd}d}|r'd}t||||||d\}}n t|||||d\}}||_||_||_ t d|_ dS) aInitialize CLAP Model Parameters ---------- enable_fusion: bool if true, it will create the fusion clap model, otherwise non-fusion clap model (default: false) device: str if None, it will automatically detect the device (gpu or cpu) amodel: str audio encoder architecture, default: HTSAT-tiny tmodel: str text encoder architecture, default: roberta Nzcuda:0cpufp32aff_2d) precisiondevice enable_fusion fusion_type)rrrz roberta-base) superr __init__torchcuda is_availablerrmodel model_cfgrfrom_pretrainedtokenize) selfrramodeltmodelrrrr __class__C/home/ubuntu/.local/lib/python3.10/site-packages/laion_clap/hook.pyrs2   zCLAP_Module.__init__cCs|j|ddddd}|S)N max_lengthTMpt)padding truncationr$return_tensors)r)rtextresultr"r"r# tokenizerAszCLAP_Module.tokenizerTc Csd}gd}|durtd|dnGtd|dkr$|jr"dnd }tjtjt}||}tj||}tj |rEtd ntd t ||tj|}td td t |dd}|j ||rdd|j D}|D]} t| d| |vrdndqzdSdS)a6Load the pretrained checkpoint of CLAP model Parameters ---------- ckpt: str if ckpt is specified, the model will load this ckpt, otherwise the model will download the ckpt from zenodo. For fusion model, it will download the 630k+audioset fusion model (id=3). For non-fusion model, it will download the 630k+audioset model (id=1). model_id: if model_id is specified, you can download our best ckpt, as: id = 0 --> 630k non-fusion ckpt id = 1 --> 630k+audioset non-fusion ckpt id = 2 --> 630k fusion ckpt id = 3 --> 630k+audioset fusion ckpt Note that if your model is specied as non-fusion model but you download a fusion model ckpt, you will face an error. z7https://huggingface.co/lukewys/laion_clap/resolve/main/)z 630k-best.ptz630k-audioset-best.ptz630k-fusion-best.ptz630k-audioset-fusion-best.ptNzLoad the specified checkpoint z from users.z&Load our best checkpoint in the paper.r-rz$The checkpoint is already downloadedz&Downloading laion_clap weight files...zDownload completed!zLoad Checkpoint...T) skip_paramscSsg|]\}}|qSr"r").0npr"r"r# usz)CLAP_Module.load_ckpt.. LoadedUnloaded)logginginforospathdirnamerealpath__file__joinexistswgetdownloadrrnamed_parameters) rckptmodel_idverbose download_linkdownload_names package_dirweight_file_name param_namesr1r"r"r# load_ckptKs0        zCLAP_Module.load_ckptc Cs|jg}|D]3}tj|dd\}}tt|}t|}i}t ||d|j r,dndd|j d|j d}| |q |j|}|sM|}|S) aget audio embeddings from the audio file list Parameters ---------- x: List[str] (N,): an audio file list to extract features, audio files can have different lengths (as we have the feature fusion machanism) use_tensor: boolean: if True, it will return the torch tensor, preserving the gradient (default: False). Returns ---------- audio_embed : numpy.darray | torch.Tensor (N,D): audio embeddings that extracted from audio files i)srSfusion rand_trunc repeatpad audio_cfgdata_truncating data_fillingrQ require_grad)revallibrosaloadrrr from_numpyfloatrrr requires_gradappendget_audio_embeddingdetachr numpy) rx use_tensor audio_inputfaudio_waveform_ temp_dict audio_embedr"r"r#!get_audio_embedding_from_filelistys&     z-CLAP_Module.get_audio_embedding_from_filelistc Cs|jg}|D],}|stt|}t|}i}t||d|jr%dndd|j d|j d}| |q |j |}|sF| }|S)aget audio embeddings from the audio data Parameters ---------- x: np.darray | torch.Tensor (N,T): audio data, must be mono audio tracks. use_tensor: boolean: if True, x should be the tensor input and the output will be the tesnor, preserving the gradient (default: False). Note that if 'use tensor' is set to True, it will not do the quantize of the audio waveform (otherwise the gradient will not be preserved). Returns ---------- audio embed: numpy.darray | torch.Tensor (N,D): audio embeddings that extracted from audio files rMrNrOrPrQrR)rrVrrrrYrZrrrr[r\r]r^r r_)rr`rarbrdrfrgr"r"r#get_audio_embedding_from_datas&     z)CLAP_Module.get_audio_embedding_from_datacCsJ|j|dur||}n||}|j|}|s#|}|S)aget text embeddings from texts Parameters ---------- x: List[str] (N,): text list tokenizer: func: the tokenizer function, if not provided (None), will use the default Roberta tokenizer. use_tensor: boolean: if True, the output will be the tesnor, preserving the gradient (default: False). Returns ---------- text_embed : numpy.darray | torch.Tensor (N,D): text embeddings that extracted from texts N)rrVr,get_text_embeddingr^r r_)rr`r,ra text_input text_embedr"r"r#rjs    zCLAP_Module.get_text_embedding)FNr r )r N)Nr-T)F)NF) __name__ __module__ __qualname__rr,rKrhrirj __classcell__r"r"r r#r s+ . %$r )__doc__r9r7rrW clap_moduler training.datarrr transformersrr@clap_module.factoryrnnModuler r"r"r"r#s