o .wiF>@sUddlmZmZmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z er9ddlmZddlmZe rVe rVddlmZddlmZdVd d Ze esUd gZnd gZe s^d gZid ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,Zeeeeeffed-<d.ed/d ed0fd1d2Z 3dWd4eeeeeeffd5fd eeeeeffd6d7Zd.ed8d9d:d;dd?Zd.ed@ed8d9d:d;dAed=eeejfd efdBdCZ DdXdEedFedGeedHe!d eeeeefff dIdJZ" K L 3dYd@ed.ed/dAed4eeeeeeffd5fd eeeeefff dMd Z#ernddNl$m%Z%ddOlm&Z&m'Z'e'e&dZ(e%e#e(dPZ)e)dQdRe)dSdRe)dTdRe)dUdRdSdS)Z) TYPE_CHECKINGLiteralUnionN)Tensor)_get_clip_model_and_processor)_SKIP_SLOW_DOCTEST_try_proceed_with_timeout)_PIQ_GREATER_EQUAL_0_8 _TRANSFORMERS_GREATER_EQUAL_4_10) CLIPModel CLIPProcessorreturncCs tjdddtjddddS)Nopenai/clip-vit-base-patch16T)resume_download) _CLIPModelfrom_pretrained_CLIPProcessorrrh/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/torchmetrics/functional/multimodal/clip_iqa.py_download_clip_for_iqa_metricsrclip_image_quality_assessmentquality)z Good photo.z Bad photo. brightness)z Bright photo.z Dark photo. noisiness)z Clean photo.z Noisy photo. colorfullness)zColorful photo.z Dull photo. sharpness)z Sharp photo.z Blurry photo.contrast)zHigh contrast photo.zLow contrast photo. complexity)zComplex photo.z Simple photo.natural)zNatural photo.zSynthetic photo.happy)z Happy photo.z Sad photo.scary)z Scary photo.zPeaceful photo.new)z New photo.z Old photo.warm)z Warm photo.z Cold photo.real)z Real photo.zAbstract photo. beautiful)zBeautiful photo.z Ugly photo.lonely)z Lonely photo.zSociable photo.relaxing)zRelaxing photo.zStressful photo._PROMPTSmodel_name_or_path)clip_iqarzopenai/clip-vit-base-patch32z!openai/clip-vit-large-patch14-336zopenai/clip-vit-large-patch14)rrcCsRddlm}|dkr%tstdddl}|jj}| d}||fSt |S)zAExtract the CLIP model and processor from the model name or path.rr r*zFor metric `clip_iqa` to work with argument `model_name_or_path` set to default value `'clip_iqa'`, package `piq` version v0.8.0 or later must be installed. Either install with `pip install piq` or`pip install torchmetrics[multimodal]`Nr) transformersr r ValueErrorpiqr*cliploadevalrr)r)rr-model processorrrr!_get_clip_iqa_model_and_processor?s  r3rprompts.cCst|ts tdg}g}d}|D]T}t|ttfstdt|tr@|tvr4tdtd|d|||t|t|trOt|dkrOtdt|tre|d||||d 7}q||fS) a}Converts the provided keywords into a list of prompts for the model to calculate the anchor vectors. Args: prompts: A string, tuple of strings or nested tuple of strings. If a single string is provided, it must be one of the available prompts (see above). Else the input is expected to be a tuple, where each element can be one of two things: either a string or a tuple of strings. If a string is provided, it must be one of the available prompts (see above). If tuple is provided, it must be of length 2 and the first string must be a positive prompt and the second string must be a negative prompt. Returns: Tuple containing a list of prompts and a list of the names of the prompts. The first list is double the length of the second list. Examples:: >>> # single prompt >>> _clip_iqa_format_prompts(("quality",)) (['Good photo.', 'Bad photo.'], ['quality']) >>> # multiple prompts >>> _clip_iqa_format_prompts(("quality", "brightness")) (['Good photo.', 'Bad photo.', 'Bright photo.', 'Dark photo.'], ['quality', 'brightness']) >>> # Custom prompts >>> _clip_iqa_format_prompts(("quality", ("Super good photo.", "Super bad photo."))) (['Good photo.', 'Bad photo.', 'Super good photo.', 'Super bad photo.'], ['quality', 'user_defined_0']) zJArgument `prompts` must be a tuple containing strings or tuples of stringsrz)All elements of `prompts` must be one of z" if not custom tuple prompts, got .zDIf a tuple is provided in argument `prompts`, it must be of length 2 user_defined_) isinstancetupler,strr(keysappendextendlen)r5 prompts_names prompts_listcountprrr_clip_iqa_format_prompts\s.     rEr1rr2rrBdevicec Cs|dkr;||d}tjt||jjtj|d}t|dD]\}}tj|tj|d||dt|f<q|| } n||ddd}| |d ||d  |} | | j d d dd S) a=Calculates the anchor vectors for the CLIP IQA metric. Args: model_name_or_path: string indicating the version of the CLIP model to use. model: The CLIP model processor: The CLIP processor prompts_list: A list of prompts device: The device to use for the calculation r*)text)dtyperF input_idsNptT)rGreturn_tensorspaddingattention_maskr7rDdimkeepdim) torchzerosr@ tokenizermodel_max_lengthlong enumeratetensor encode_textfloatget_text_featurestonorm) r)r1r2rBrFtext_processed anchors_textitpanchorsrrr_clip_iqa_get_anchor_vectorss $rcimages data_rangec Cs|t|} |dkr;tjgd|ddddd}tjgd|ddddd}|||}|j|dd}n|d d |Dd d d } || d|}||jddd dS)Nr*)g3<4'?gwg M?gy{ ?)rFr9)gB91?gwt.?g U?F) pos_embeddingcSsg|]}|qSr)cpu).0r`rrr sz$_clip_iqa_update..rJT)rdrKrL pixel_valuesr7rNrO)rZrRrXview encode_imageget_image_featuresr\r]) r)rdr1r2rerF default_mean default_std img_featuresprocessed_inputrrr_clip_iqa_updates  rsTrqrbrAformat_as_dictcsnd||}||jdddddddddft|dkr(S|r5fddt|DSS) zFinal computation of CLIP IQA.drrNr7Nr9cs"i|] \}}|dd|fqS)Nr)rir`rDprobsrr s"z%_clip_iqa_compute..)treshapeshapesoftmaxr@squeezerW)rqrbrArtlogits_per_imagerrvr_clip_iqa_computes, rr*?c Cst|\}}t|\}}|j}||}tt|||||} t||||||} t| | |WdS1s:wYdS)aCalculates `CLIP-IQA`_, that can be used to measure the visual content of images. The metric is based on the `CLIP`_ model, which is a neural network trained on a variety of (image, text) pairs to be able to generate a vector representation of the image and the text that is similar if the image and text are semantically similar. The metric works by calculating the cosine similarity between user provided images and pre-defined prompts. The prompts always come in pairs of "positive" and "negative" such as "Good photo." and "Bad photo.". By calculating the similartity between image embeddings and both the "positive" and "negative" prompt, the metric can determine which prompt the image is more similar to. The metric then returns the probability that the image is more similar to the first prompt than the second prompt. Build in prompts are: * quality: "Good photo." vs "Bad photo." * brightness: "Bright photo." vs "Dark photo." * noisiness: "Clean photo." vs "Noisy photo." * colorfullness: "Colorful photo." vs "Dull photo." * sharpness: "Sharp photo." vs "Blurry photo." * contrast: "High contrast photo." vs "Low contrast photo." * complexity: "Complex photo." vs "Simple photo." * natural: "Natural photo." vs "Synthetic photo." * happy: "Happy photo." vs "Sad photo." * scary: "Scary photo." vs "Peaceful photo." * new: "New photo." vs "Old photo." * warm: "Warm photo." vs "Cold photo." * real: "Real photo." vs "Abstract photo." * beautiful: "Beautiful photo." vs "Ugly photo." * lonely: "Lonely photo." vs "Sociable photo." * relaxing: "Relaxing photo." vs "Stressful photo." Args: images: Either a single ``[N, C, H, W]`` tensor or a list of ``[C, H, W]`` tensors model_name_or_path: string indicating the version of the CLIP model to use. By default this argument is set to ``clip_iqa`` which corresponds to the model used in the original paper. Other available models are `"openai/clip-vit-base-patch16"`, `"openai/clip-vit-base-patch32"`, `"openai/clip-vit-large-patch14-336"` and `"openai/clip-vit-large-patch14"` data_range: The maximum value of the input tensor. For example, if the input images are in range [0, 255], data_range should be 255. The images are normalized by this value. prompts: A string, tuple of strings or nested tuple of strings. If a single string is provided, it must be one of the available prompts (see above). Else the input is expected to be a tuple, where each element can be one of two things: either a string or a tuple of strings. If a string is provided, it must be one of the available prompts (see above). If tuple is provided, it must be of length 2 and the first string must be a positive prompt and the second string must be a negative prompt. .. hint:: If using the default `clip_iqa` model, the package `piq` must be installed. Either install with `pip install piq` or `pip install torchmetrics[multimodal]`. Returns: A tensor of shape ``(N,)`` if a single prompts is provided. If a list of prompts is provided, a dictionary of with the prompts as keys and tensors of shape ``(N,)`` as values. Raises: ModuleNotFoundError: If transformers package is not installed or version is lower than 4.10.0 ValueError: If not all images have format [C, H, W] ValueError: If prompts is a tuple and it is not of length 2 ValueError: If prompts is a string and it is not one of the available prompts ValueError: If prompts is a list of strings and not all strings are one of the available prompts Example:: Single prompt: >>> from torch import randint >>> from torchmetrics.functional.multimodal import clip_image_quality_assessment >>> imgs = randint(255, (2, 3, 224, 224)).float() >>> clip_image_quality_assessment(imgs, prompts=("quality",)) tensor([0.8894, 0.8902]) Example:: Multiple prompts: >>> from torch import randint >>> from torchmetrics.functional.multimodal import clip_image_quality_assessment >>> imgs = randint(255, (2, 3, 224, 224)).float() >>> clip_image_quality_assessment(imgs, prompts=("quality", "brightness")) {'quality': tensor([0.8693, 0.8705]), 'brightness': tensor([0.5722, 0.4762])} Example:: Custom prompts. Must always be a tuple of length 2, with a positive and negative prompt. >>> from torch import rand >>> from torchmetrics.functional.multimodal import clip_image_quality_assessment >>> imgs = randint(255, (2, 3, 224, 224)).float() >>> clip_image_quality_assessment(imgs, prompts=(("Super good photo.", "Super bad photo."), "brightness")) {'user_defined_0': tensor([0.9578, 0.9654]), 'brightness': tensor([0.5495, 0.5764])} N) rEr3rFr\rRinference_modercrsr) rdr)rer5rBrAr1r2rFrbrqrrrrs h    $)partial)Anycast)rd)r)r5)rrr)rrrr))zPhoto of a catzPhoto of a dogr)zColorful photozBlack and white photo)rN)r4)T)r*rr4)*typingrrrrRr-torchmetrics.functional.multimodal.clip_scorertorchmetrics.utilities.checksrrtorchmetrics.utilities.importsr r r+r rr rr__doctest_skip__r(dictr<r;__annotations__r3listrErFrcrZrsboolrr functoolsrrrrdfrrrrs               7  "      t