o Gi@snddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl ZddlZeedddurTddlmZmZddlmZddlmZd d lmZd d lmZd d lmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&e&rddl'Z'e'j(j)*rddl)Z)e"rddl+m,Z,e#rddl-m.Z.e%rddl/m0Z0e$rddl1Z1de2fddZ3ddZ4de5fddZ6 dVdededej7dej7dej7dej7d ej7d!e8d"e9ej7ej7ffd#d$Z:ded"e;e5ej7ffd%d&Zj?e@ej>j?Bfd(d)ZAd*e;e5ej7fd+e5d,ej>j?fd-d.ZBd/e;e5ej>j?fd"e;e5e ffd0d1ZC    2 dWd3e5d4e2d5e8d6e8d7e8d8ejDe5Bd9ejEdBfd:d;ZFdXd3e5fdd?ZHed@dAdBej>j?eBd8e5ejDBdCeIfdDdEZJdFdGZKdHdIZLd"e;fdJdKZMd"e;fdLdMZN @ @ @  dYd'ej>j?d8e5ejDBdCeIdNeIdOeIdPe;e5e fdBdQeOePej>j?dBd"efdRdSZQGdTdUdUZRdS)ZN)contextmanager)partial)AnyIterable distributed) CPUOffloadShardingStrategy)FullyShardedDataParallel)transformer_auto_wrap_policy)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_accelerate_availableis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available) get_logger)set_peft_model_state_dict) transformsseedcCsFt|tj|t|trtj|dStj|dS)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rr"L/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/training_utils.pyset_seed7s  r$cCs|j}|d}d|d}|j|jd|}t|jt|jkr0|d}t|jt|jks"||j}|j|jd|}t|jt|jkrY|d}t|jt|jksK||j}||d}|S)a Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor(floatlenshapeexpand)noise_scheduler timestepsr*sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrr"r"r# compute_snrKs    r7interpolation_typecCststd|dkrtjj}|S|dkrtjj}|S|dkr%tjj}|S|dkr/tjj}|S|dkr9tjj}|S|dkrCtjj }|S|dkrMtjj }|St d |d ) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r8interpolation_moder"r"r#resolve_interpolation_modeps8    rKr&unetr0r1noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncCs|j|j|dddf}d|d} | |} d} t||||j} Wdn1s/wYd\} } |jjdkr[| }||}| | | | |} | |} | | fS|jjdkret dt d|jj) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr&r%)NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r*r+r(rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrI)rLr0r1rMrNrOrPrQr*r3 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noiser"r"r# compute_dream_and_update_latentss$       rccCsbi}|D](\}}t|dr.t|d}|dur.|}|D] \}}|||d|<q q|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerNz.lora.) named_moduleshasattrgetattr state_dictitems)rLlora_state_dictnamemodulerecurrent_lora_layer_sdlora_layer_matrix_name lora_paramr"r"r#unet_lora_state_dicts  rqmodelcCs>t|ts|g}|D]}|D] }|jr|||_qq dS)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr+data)rrdtypemparamr"r"r#cast_training_paramss   r{rkprefix text_encodercs4fdd|D}tt|}t||dddS)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. cs*i|]\}}|r|d|qS)) startswithreplace.0kvr|r"r# s z5_set_state_dict_into_text_encoder..default) adapter_nameN)rjrrr)rkr|r}text_encoder_state_dictr"rr#!_set_state_dict_into_text_encoders  rmodules_to_savecCs:i}|D]\}}|dur|jd||d<q|S)Nr_lora_adapter_metadata)rj peft_configto_dict)r metadatas module_namermr"r"r#_collate_lora_metadata s rcpuweighting_scheme batch_size logit_mean logit_std mode_scaler( generatorcCs|dkrtj|||f||d}tjj|}|S|dkr=tj|f||d}d||ttj|ddd|}|Stj|f||d}|S)a Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer(rmode)rr(rr r)) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr(rur"r"r#%compute_density_for_timestep_samplings,rcCsX|dkr |d}|S|dkr%dd|d|d}dtj|}|St|}|S)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r))r,rrr ones_like)rsigmas weightingbotr"r"r#compute_loss_weighting_for_sd3,s  rcCsxttjrtjdStjjrtjdStr't j dSt tdr8tj r:tj dSdSdS)zV Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr! is_available empty_cachebackendsmpsr torch_npurrgrr"r"r"r# free_memory>s  rT)offloadmodulesrc gs|r/tdd|D }|rdd|D}nt|dksJ|djg}|D]}||q'zdVW|rEt||D] \}}||q;dSdS|rZt||D] \}}||qPww)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. css|]}t|tVqdSN)rsr rryr"r"r# Ysz!offload_models..cSsg|] }t|jqSr")nextrur(rr"r"r# \z"offload_models..r rN)anyr-r(r+zip)r(rris_modeloriginal_devicesryorig_devr"r"r#offload_modelsNs*     rc Cs|std|d}g}|D]d}td|}|s#td|dz;t|d}t|d}|dks:|dkr>td |d dksJ|d dkrVtd |d |d | ||fWqtyu}z td|d||d}~ww|s|td|S)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r)rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rIstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidther"r"r#parse_buckets_stringns. rc CsJtd}d}t|D]\}\}}t||||}||kr"|}|}q |S)z6Finds the closes bucket to the given height and width.infN)r, enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetricr"r"r#find_nearest_bucketsrcCsdd|DS)NcSs2i|]\}}|t|tjr|n|qSr")rsrTensorrYr contiguousrr"r"r#rs2z&_to_cpu_contiguous..)rj) state_dictsr"r"r#_to_cpu_contiguoussrcCsTi}t|jdd}|durtd|jj}|dur tj|d<|S|jp%tj|d<|S)zT Extract and convert FSDP config from Accelerator into PyTorch FSDP kwargs. fsdp_pluginNzLAccelerate isn't configured to handle FSDP. Please update your installation.sharding_strategy)rhstaterIrr FULL_SHARDr) acceleratorkwargs fsdp_staterr"r"r# get_fsdp_kwargs_from_accelerators ruse_orig_paramslimit_all_gathers fsdp_kwargstransformer_layer_clsc Cstt}|durt|jjjd}|d|jtt|hd}||r)t |dnd|||d} |r7| |t |fi| } | S)uY Wrap a model with FSDP using common defaults and optional transformer auto-wrapping. Args: model: Model to wrap device: Target device (e.g., accelerator.device) offload: Whether to enable CPU parameter offloading use_orig_params: Whether to use original parameters limit_all_gathers: Whether to limit all gathers fsdp_kwargs: FSDP arguments (sharding_strategy, etc.) — usually from Accelerate config transformer_layer_cls: Classes for auto-wrapping (if not using policy from fsdp_kwargs) Returns: FSDP-wrapped model Nrz8transformer_layer_cls is not provided, auto-inferred as )r)offload_params) device_id cpu_offloadrrauto_wrap_policy) r__name__typerrlanguage_modellayersinforr rupdateFSDP) rrr(rrrrrloggerrrW fsdp_modelr"r"r#wrap_with_fsdps rc@sBeZdZdZ         d.d eejjd ed ed e d e dee Bdee Bde de dBde e e fdBfddZed/d0ddZddZde defddZed eejjfddZd eejjddfdd Zd1d!d"Zd2d1d#d$Zde fd%d&Zd eejjddfd'd(Zd eejjddfd)d*Zd+e ddfd,d-ZdS)3EMAModelz6 Exponential Moving Average of models weights H.?rFr&UUUUUU?Nrudecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc Kst|tjjrd} tdd| dd|}d}| dddur-d } tdd| dd| d}| d ddurCd } td d| dd| d }t|}d d |D|_| dddurid} tdd| dd|j | ddd|_ ||_ ||_ ||_ ||_||_||_d|_d|_||_| |_| |_dS)ar Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (str | torch.device | None): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.cSsg|]}|qSr")clonerYrpr"r"r#r'sz%EMAModel.__init__..r(z=The `device` argument is deprecated. Please use `to` instead.r'r)rsrrModulerrugetrt shadow_paramsr+temp_stored_paramsr r r r r roptimization_stepcur_decay_valuerrr) selfrur r r r r rrrrrdeprecation_messager"r"r#__init__sJ# zEMAModel.__init__rRcCs@|j|dd\}}||}||||j|d}|||S)NT)return_unused_kwargs)rrr) from_configfrom_pretrainedrurWload_state_dict)clspathrr_ ema_kwargsrr ema_modelr"r"r#r%=s   zEMAModel.from_pretrainedcCsr|jdur td|jdurtd|j|j}|}|dd|jdi|||| |dS)NzJ`save_pretrained` can only be used if `model_cls` was defined at __init__.zM`save_pretrained` can only be used if `model_config` was defined at __init__.rr") rrIrr$ripopregister_to_configcopy_torusave_pretrained)r r(rrrir"r"r#r/Gs   zEMAModel.save_pretrainedrcCsntd||jd}|dkrdS|jr!dd||j|j }nd|d|}t||j}t||j}|S)zN Compute the decay factor for the exponential moving average. rr r )maxr r r rminr r )r rsteprr"r"r# get_decayVs  zEMAModel.get_decayc Cst|tjjrd}tdd|dd|}t|}|jd7_||j}||_ d|}t }|j rt rEtjjrEtjj|dd}|Edd |D}d d t|j|D}t|t|krxtjd d t|j|Dd d |Dd dtj|t|||dWddS1swYdSt|j|D]9\}} t rtjjrtjj| dd}|| jr|||| n|| Wdn1swYqdS)NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr ) modifier_rankcSsg|]}|jr|qSr"rvrrzr"r"r#rz!EMAModel.step..cSsg|] \}}|jr|qSr"r6rs_paramrzr"r"r#rs  cSsg|] \}}|js|qSr"r6r9r"r"r#rrcSsg|]}|js|qSr"r6r7r"r"r#rr8T) non_blocking)r4)rsrrrrrurtrr4r contextlib nullcontextrr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrrr-_foreach_copy_ _foreach_sub_ _foreach_subrvsub_copy_) r rur!r one_minus_decaycontext_manager params_grad s_params_gradr:rzr"r"r#r3isZ   " z EMAModel.stepcCsjt|}|jrtdd|Dddt|j|DdSt|j|D]\}}|j||j jq#dS)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. cSg|]}|jqSr"rwr7r"r"r#rz$EMAModel.copy_to..cSsg|] \}}||jjqSr")r+r(rwr9r"r"r#rsN) rtrrrDrrrwrHr+r()r rur:rzr"r"r#r.s  zEMAModel.copy_tocCsdd|jD|_dS)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. cSsg|]}|qSr") pin_memoryrr"r"r#rsz'EMAModel.pin_memory..Nrr r"r"r#rPszEMAModel.pin_memorycsfdd|jD|_dS)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` cs2g|]}|r|jdn|jdqS)r(rxr;)r(r;)is_floating_pointr+rrSr"r#rs  zEMAModel.to..NrQ)r r(rxr;r"rSr#r+s z EMAModel.toc Cs&|j|j|j|j|j|j|j|jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. r r rr r r rrrUrRr"r"r#ris zEMAModel.state_dictcCsdd|D|_dS)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. cSsg|] }|qSr")rYrrr7r"r"r#rsz"EMAModel.store..N)r)r rur"r"r#storeszEMAModel.storecCsj|jdur td|jrtdd|Ddd|jDnt|j|D] \}}|j|jq$d|_dS)aG Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`cSrMr"rNr7r"r"r#rrOz$EMAModel.restore..cSrMr"rN)rc_paramr"r"r#rrO)r RuntimeErrorrrrDrrwrH)r rurWrzr"r"r#restores  zEMAModel.restorericCsht|}|d|j|_|jdks|jdkrtd|d|j|_t|jts-td|d|j|_t|jt s?td|d |j |_ t|j t sQtd |d |j |_ t|j t sctd |d |j |_ t|j tt fswtd|d|j|_t|jtt fstd|dd}|dur||_t|jtstdtdd|jDstddSdS)a Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. r rr&zDecay must be between 0 and 1r zInvalid min_decayrzInvalid optimization_stepr zInvalid update_after_stepr zInvalid use_ema_warmupr zInvalid inv_gammarz Invalid powerrNzshadow_params must be a listcss|] }t|tjVqdSr)rsrrrr"r"r#r/sz+EMAModel.load_state_dict..z!shadow_params must all be Tensors)copydeepcopyrr rIr rsr,rrr r boolr rrrtall)r rirr"r"r#r&s>       zEMAModel.load_state_dict) rrrFr&rFNN)F)rRr)rRN)NNF)r __module__ __qualname____doc__rrr Parameterr,rr\rdictstrr" classmethodr%r/r4rUr3r.rPr+rirVrYr&r"r"r"r#rsZ    T 9   r)r&)NNNrNr)TTTNN)Sr<rZrrrrrr functoolsrtypingrrnumpyrrrhtorch.distributed.fsdprrr rtorch.distributed.fsdp.wrapr modelsr pipelinesr schedulersrutilsrrrrrrrrr>r?r@rAaccelerate.loggingrpeftr torchvisionrrrr$r7rcrKrr,tuplercrbrqfloat32rrrtr{rrr( Generatorrrrr\rrrrrsetrrrr"r"r"r#s       (     %4 6$  &  ,  3