o Gi8@sPddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z mZddlmZmZeeZejdduZejdduZejd duZer[ddlZdd lmZned erpdd lmZdd lm Z m!Z!nederddl"m#Z#ddl$m%Z%nedd ddZ&d!ddZ'ddZ(  d"ddZ) d#ddZ*ddZ+dS)$N)ImageImageOps)InterpolationMode) normalizeresize) get_logger load_image insightfaceconsisid_eva_clipfacexlib) FaceAnalysiszPinsightface is not available. Please install it using 'pip install insightface'.)create_model_and_transforms)OPENAI_DATASET_MEANOPENAI_DATASET_STDz\consisid_eva_clip is not available. Please install it using 'pip install consisid_eva_clip'.)init_parsing_model)FaceRestoreHelperzJfacexlib is not available. Please install it using 'pip install facexlib'.cCsd|jdd\}}t|||kr|S|t||}t||}t||}tj|||ftjd}|S)a Resize the input image to a specified long edge while maintaining aspect ratio. Args: image (numpy.ndarray): Input image (H x W x C or H x W). resize_long_edge (int): The target size for the long edge of the image. Default is 768. Returns: numpy.ndarray: Resized image with the long edge matching `resize_long_edge`, while maintaining the aspect ratio. N) interpolation)shapemaxintcv2rINTER_LANCZOS4)imageresize_long_edgehwkr _/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/consisid/consisid_utils.pyresize_numpy_image_long's   r"Tcs4ddt|trfdd|DS|S)aONumpy array to tensor. Args: imgs (list[ndarray] | ndarray): Input images. bgr2rgb (bool): Whether to change bgr to rgb. float32 (bool): Whether to change to float32. Returns: list[tensor] | tensor: Tensor images. If returned results only have one element, just return tensor. cSsX|jddkr|r|jdkr|d}t|tj}t|ddd}|r*| }|S)Nrrfloat64float32r) rdtypeastypercvtColor COLOR_BGR2RGBtorch from_numpy transposefloat)imgbgr2rgbr$r r r! _totensorKs  zimg2tensor.._totensorcsg|]}|qSr r ).0r.r0r/r$r r! Vszimg2tensor..) isinstancelist)imgsr/r$r r2r! img2tensor>s  r7cCs\d|ddddfd|ddddfd|ddddf}|dddd}|S) a Converts an RGB image to grayscale by applying the standard luminosity formula. Args: img (torch.Tensor): The input image tensor with shape (batch_size, channels, height, width). The image is expected to be in RGB format (3 channels). Returns: torch.Tensor: The grayscale image tensor with shape (batch_size, 3, height, width). The grayscale values are replicated across all three channels. gA`"?Nrr%gbX9?rgv/?r)repeat)r.xr r r!to_grayZsH r:c s0|t|tj} || } t| dkr)t| dddd} | d} | d}nd} d}|| |jd d |durA|j d}| t|j dkrPt d |j d}| durct d ||} t| ||} | jd krv| d} | rt|d ddd}||}|t|gdgddjd d dgd}tfdd|D}t|}t||t|}t|||}nt| tj}t|d ddd}||}|}}t||jt j!}t|||}|||dd dd\}}t"|dd d }t#||}tj$| |gdd}||||fS)a Process face embeddings from an image, extracting relevant features such as face embeddings, landmarks, and parsed face features using a series of face detection and alignment tools. Args: face_helper_1: Face helper object (first helper) for alignment and landmark detection. clip_vision_model: Pre-trained CLIP vision model used for feature extraction. face_helper_2: Face helper object (second helper) for embedding extraction. eva_transform_mean: Mean values for image normalization before passing to EVA model. eva_transform_std: Standard deviation values for image normalization before passing to EVA model. app: Application instance used for face detection. device: Device (CPU or GPU) where the computations will be performed. weight_dtype: Data type of the weights for precision (e.g., `torch.float32`). image: Input image in RGB format with pixel values in the range [0, 255]. original_id_image: (Optional) Original image for feature extraction if `is_align_face` is False. is_align_face: Boolean flag indicating whether face alignment should be performed. Returns: tuple: - id_cond: Concatenated tensor of Ante face embedding and CLIP vision embedding - id_vit_hidden: Hidden state of the CLIP vision model, a list of tensors. - return_face_features_image_2: Processed face features image after normalization and parsing. - face_kps: Keypoints of the face detected in the image. rcSs0|dd|dd|dd|ddS)Nbboxrrrr%r )r9r r r!s0z)process_face_embeddings..)key embeddingkpsNT)only_center_facezfacexlib align face failzMFailed to detect face using insightface. Extracting embedding with align facer%)r/go@)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?)dimkeepdim)r c3s|]}|kVqdS)Nr )r1i parsing_outr r! sz*process_face_embeddings..F)return_all_features return_hiddenshuffler)rB)% clean_allrr( COLOR_RGB2BGRgetlensorted read_imageget_face_landmarks_5all_landmarks_5align_warp_face cropped_faces RuntimeErrorloggerwarningget_featr*r+tondim unsqueezer7 face_parserargmaxsumbool ones_likewherer:r image_sizerBICUBICnormdivcat) face_helper_1clip_vision_model face_helper_2eva_transform_meaneva_transform_stdappdevice weight_dtyperoriginal_id_image is_align_face image_bgr face_infoid_ante_embeddingface_kps align_faceinputbg_labelbg white_imagereturn_face_features_imagereturn_face_features_image_2original_image_bgrface_features_image id_cond_vit id_vit_hiddenid_cond_vit_normid_condr rLr!process_face_embeddingsksp&                  rc  Cst|trtt|dd} nttt |d} t | d} | } t ||||||||| | | \} } }}| }|}|ddd}|d}|tj}tt |} | | | |fS)a Process face embeddings from an input image for inference, including alignment, feature extraction, and embedding concatenation. Args: face_helper_1: Face helper object (first helper) for alignment and landmark detection. clip_vision_model: Pre-trained CLIP vision model used for feature extraction. face_helper_2: Face helper object (second helper) for embedding extraction. eva_transform_mean: Mean values for image normalization before passing to EVA model. eva_transform_std: Standard deviation values for image normalization before passing to EVA model. app: Application instance used for face detection. device: Device (CPU or GPU) where the computations will be performed. weight_dtype: Data type of the weights for precision (e.g., `torch.float32`). img_file_path: Path to the input image file (string) or a numpy array representing an image. is_align_face: Boolean flag indicating whether face alignment should be performed (default: True). Returns: tuple: - id_cond: Concatenated tensor of Ante face embedding and CLIP vision embedding. - id_vit_hidden: Hidden state of the CLIP vision model, a list of tensors. - image: Processed face image after feature extraction and alignment. - face_kps: Keypoints of the face detected in the image. )rRGBir%rr)r4strnparrayr convertrexif_transposer fromarrayr"rcpudetachsqueezepermutenumpyr'uint8)rnrorprqrrrsrtru img_file_pathrwrrvrralign_crop_face_imager{tensorr r r!process_face_embeddings_infers2 %     rc CsTtddddd|tj|dd}d|_td |tj|dd |_tjj|d d gd }|j ddt dtj|dddd\}}}|j }t |dt }t |dt} t|ttfs]|fd}t| ttfsi| fd} |}| } tdtj|dd gd} | j ddd|j|j||j||j||j||d|||| || fS)a Prepare all face models for the facial recognition task. Parameters: - model_path: Path to the directory containing model files. - device: The device (e.g., 'cuda', 'xpu', 'cpu') where models will be loaded. - dtype: Data type (e.g., torch.float32) for model inference. Returns: - face_helper_1: First face restoration helper. - face_helper_2: Second face restoration helper. - face_clip_model: CLIP model for face extraction. - eva_transform_mean: Mean value for image normalization. - eva_transform_std: Standard deviation value for image normalization. - face_main_model: Main face analysis model. r%i)r%r%retinaface_resnet50png face_encoder)upscale_factor face_size crop_ratio det_modelsave_extrtmodel_rootpathNbisenet) model_namertrz./face_encoder/models/antelopev2/glintr100.onnxCUDAExecutionProvider) providersr)ctx_idzEVA02-CLIP-L-14-336zEVA02_CLIP_L_336_psz14_s6B.ptT)force_custom_clip image_mean image_stdr antelopev2)namerootr)r)rdet_size)r&)rospathjoinrcrr model_zoo get_modelpreparervisualgetattrrrr4r5tupler face_detevalr`) model_pathrtr&rnrpmodel_face_clip_modelrqrrface_main_modelr r r!prepare_face_models#sT             r)r)TT)NT)T),importlib.util importlibrrrrr*PILrrtorchvision.transformsr!torchvision.transforms.functionalrrutilsrr __name__r]util find_spec_insightface_available_consisid_eva_clip_available_facexlib_availabler insightface.appr ImportErrorr rconsisid_eva_clip.constantsrrfacexlib.parsingr&facexlib.utils.face_restoration_helperrr"r7r:rrrr r r r!sD      z H