o pib@sddlZddlZddlZddlZddlmZmZmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZddlmZmZmZmZmZmZmZerOddlZerXddlmZeraddlmZerhddl Z d e!fd d Z"d d Z#de$fddZ% d2dededej&dej&dej&dej&dej&de'de e ej&e ej&ffddZ(dedee$ej&ffddZ)ej*fde ej+j,eej+j,ffd d!Z-d"ee$ej&fd#e$d$ej+j,fd%d&Z. d3d'e$d(e!d)e'd*e'd+e'f d,d-Z/d4d'e$fd.d/Z0Gd0d1d1Z1dS)5N)AnyDictIterableListOptionalTupleUnion)UNet2DConditionModel)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available)set_peft_model_state_dict) transformsseedcCsFt|tj|t|trtj|dStj|dS)z Args: Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. seed (`int`): The seed to set. N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rrV/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/training_utils.pyset_seed$s   rcCs|j}|d}d|d}|j|jd|}t|jt|jkr0|d}t|jt|jks"||j}|j|jd|}t|jt|jkrY|d}t|jt|jksK||j}||d}|S)z Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 ??device).N)alphas_cumprodtor#floatlenshapeexpand)noise_scheduler timestepsr%sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrrrr compute_snr4s    r2interpolation_typecCststd|dkrtjj}|S|dkrtjj}|S|dkr%tjj}|S|dkr/tjj}|S|dkr9tjj}|S|dkrCtjj }|S|dkrMtjj }|St d |d ) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r3interpolation_moderrrresolve_interpolation_modeNs8    rFr!unetr+r,noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncCs|j|j|dddf}d|d} | |} d} t||||j} Wdn1s/wYd\} } |jjdkr[| }||}| | | | |} | |} | | fS|jjdkret dt d|jj) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from http://arxiv.org/abs/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr!r )NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r%r&r#rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrD)rGr+r,rHrIrJrKrLr%r. dream_lambdapred_noisy_latents_targetpredicted_noise delta_noiserrr compute_dream_and_update_latentszs$       r^cCsbi}|D](\}}t|dr.t|d}|dur.|}|D] \}}|||d|<q q|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerNz.lora.) named_moduleshasattrgetattr state_dictitems)rGlora_state_dictnamemoduler`current_lora_layer_sdlora_layer_matrix_name lora_paramrrrunet_lora_state_dicts  rlmodelcCs>t|ts|g}|D]}|D] }|jr|||_qq dSN) isinstancelist parameters requires_gradr&data)rmdtypemparamrrrcast_training_paramss   rwrfprefix text_encodercs4fdd|D}tt|}t||dddS)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. cs*i|]\}}|r|d|qS)) startswithreplace).0kvrxrr s z5_set_state_dict_into_text_encoder..default) adapter_nameN)rer r r)rfrxrytext_encoder_state_dictrrr!_set_state_dict_into_text_encoders  rweighting_scheme batch_size logit_mean logit_std mode_scalecCs|dkrtj|||fdd}tjj|}|S|dkr;tj|fdd}d||ttj|ddd|}|Stj|fdd}|S)zCompute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://arxiv.org/abs/2403.03206v1. logit_normalcpu)meanstdsizer#mode)rr#r r$) rnormalnn functionalsigmoidrandcosmathpi)rrrrrurrr%compute_density_for_timestep_samplings ,rcCsX|dkr |d}|S|dkr%dd|d|d}dtj|}|St|}|S)zComputes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://arxiv.org/abs/2403.03206v1. sigma_sqrtgcosmapr r$)r'rrr ones_like)rsigmas weightingbotrrrcompute_loss_weighting_for_sd3s  rc@sFeZdZdZ         d.d eejjd ed ed e d e de ee fde ee fde de e deee ffddZed/d0ddZddZde defddZed eejjfddZd eejjddfdd Zd1d!d"Zd2d1d#d$Zdefd%d&Zd eejjddfd'd(Zd eejjddfd)d*Zd+eddfd,d-ZdS)3EMAModelz6 Exponential Moving Average of models weights H.?rFr!UUUUUU?Nrqdecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc Kst|tjjrd} tdd| dd|}d}| dddur-d } tdd| dd| d}| d ddurCd } td d| dd| d }t|}d d |D|_| dddurid} tdd| dd|j | ddd|_ ||_ ||_ ||_ ||_||_||_d|_d|_||_| |_| |_dS)a{ Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.cSsg|]}|qSr)clonerTr}prrr Jsz%EMAModel.__init__..r#z=The `device` argument is deprecated. Please use `to` instead.r"r)rorrModulerrqgetrp shadow_paramsr&temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerrr) selfrqrrrrrrrrrkwargsdeprecation_messagerrr__init__ sJ# zEMAModel.__init__rMcCs@|j|dd\}}||}||||j|d}|||S)NT)return_unused_kwargs)rrr) load_configfrom_pretrainedrqrRload_state_dict)clspathrr_ ema_kwargsrm ema_modelrrrr`s   zEMAModel.from_pretrainedcCsr|jdur td|jdurtd|j|j}|}|dd|jdi|||| |dS)NzJ`save_pretrained` can only be used if `model_cls` was defined at __init__.zM`save_pretrained` can only be used if `model_config` was defined at __init__.rr) rrDr from_configrdpopregister_to_configcopy_torqsave_pretrained)rrrmrdrrrrjs   zEMAModel.save_pretrainedrcCsntd||jd}|dkrdS|jr!dd||j|j }nd|d|}t||j}t||j}|S)zN Compute the decay factor for the exponential moving average. rr r )maxrrrrminrr)rrsteprrrr get_decayys  zEMAModel.get_decayc Cst|tjjrd}tdd|dd|}t|}|jd7_||j}||_ d|}t j }t rpassing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr r) modifier_rankcSsg|]}|jr|qSrrrr}rvrrrrz!EMAModel.step..cSsg|] \}}|jr|qSrrr}s_paramrvrrrrs  cSsg|] \}}|js|qSrrrrrrrscSsg|]}|js|qSrrrrrrrrT) non_blocking)r/)rorrrrrqrprrr contextlib nullcontextr transformers deepspeedis_deepspeed_zero3_enabledrzeroGatheredParametersziprr(_foreach_copy_ _foreach_sub_ _foreach_subrrsub_copy_) rrqrrone_minus_decaycontext_managerr params_grad s_params_gradrrvrrrrs^   " z EMAModel.stepcCsjt|}|jrtdd|Dddt|j|DdSt|j|D]\}}|j||j jq#dS)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. cSg|]}|jqSrrsrrrrrz$EMAModel.copy_to..cSsg|] \}}||jjqSr)r&r#rsrrrrrsN) rprrrrrrsrr&r#)rrqrrvrrrrs  zEMAModel.copy_tocCsdd|jD|_dS)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. cSsg|]}|qSr) pin_memoryrrrrrsz'EMAModel.pin_memory..NrrrrrrszEMAModel.pin_memorycsfdd|jD|_dS)zMove internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` cs2g|]}|r|jdn|jdqS)r#rtr)r#r)is_floating_pointr&rrrrrs  zEMAModel.to..Nr)rr#rtrrrrr&s z EMAModel.toc Cs&|j|j|j|j|j|j|j|jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrrrrrrrrrrrrrds zEMAModel.state_dictcCsdd|D|_dS)z Args: Save the current parameters for restoring later. parameters: Iterable of `torch.nn.Parameter`; the parameters to be temporarily stored. cSsg|] }|qSr)rTrrrrrrr sz"EMAModel.store..N)r)rrqrrrstoreszEMAModel.storecCsj|jdur td|jrtdd|Ddd|jDnt|j|D] \}}|j|jq$d|_dS)aF Args: Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`cSrrrrrrrrrz$EMAModel.restore..cSrrr)r}c_paramrrrrr)r RuntimeErrorrrrrrsr)rrqrrvrrrrestore s  zEMAModel.restorerdcCsht|}|d|j|_|jdks|jdkrtd|d|j|_t|jts-td|d|j|_t|jt s?td|d |j |_ t|j t sQtd |d |j |_ t|j t sctd |d |j |_ t|j tt fswtd|d|j|_t|jtt fstd|dd}|dur||_t|jtstdtdd|jDstddSdS)a Args: Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rrr!zDecay must be between 0 and 1rzInvalid min_decayrzInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerrNzshadow_params must be a listcss|] }t|tjVqdSrn)rorTensorrrrr Psz+EMAModel.load_state_dict..z!shadow_params must all be Tensors)copydeepcopyrrrDrror'rintrrboolrrrrpall)rrdrrrrr$s>       zEMAModel.load_state_dict) rrrFr!rFNN)F)rMr)rMN)NNF)__name__ __module__ __qualname____doc__rrr Parameterr'rrrrrrstrr classmethodrrrrPrrrr&dictrdrrrrrrrrsZ      T ;   r)r!)NNNrn)2rrrrtypingrrrrrrrnumpyrrmodelsr schedulersr utilsr r rrrrrrpeftr torchvisionr torch_npurrr2rrFrr'r^rlfloat32rrrwrrrrrrrrs$  $   4 6(