o piI@sddlmZmZmZddlZddlmmZddlmZddl m Z m Z ddl m Z e r6ddlmZmZe eZdd Zd d Zdd efddZGdddejjZGdddejZGdddejZGdddejZGdddejZdS))OptionalTupleUnionN)nn) deprecatelogging)is_transformers_available) CLIPTextModelCLIPTextModelWithProjectioncCs`g}t|ttfr't|jjjD]\}}d|d}|j}|||fq|St d|j j )Ntext_model.encoder.layers.z .self_attnz.do not know how to get attention modules for: ) isinstancer r enumerate text_modelencoderlayers self_attnappend ValueError __class____name__) text_encoder attn_modulesilayernamemodrS/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/lora.pytext_encoder_attn_modules)s rcCs`g}t|ttfr't|jjjD]\}}|j}d|d}|||fq|St d|j j )Nr z.mlpz(do not know how to get mlp modules for: ) r r r rrrrmlprrrr)r mlp_modulesrrmlp_modrrrrtext_encoder_mlp_modules7s r#? lora_scalecCspt|D]\}}t|jtr||j_||j_||j_||j_qt|D]\}}t|j tr5||j _||j _q#dSN) rr q_projPatchedLoraProjectionr%k_projv_projout_projr#fc1fc2)rr%_ attn_module mlp_modulerrradjust_lora_scale_text_encoderEs  r1csNeZdZdfdd Zddddfd d Zdd d ZddZddZZS)r(Nc spd}tdd|tddlm}||_|jjj}|dur$|jjj}||jj |jj ||||d|_ ||_ dS)NzsUse of `PatchedLoraProjection` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.r(1.0.0r)LoRALinearLayer) network_alphadevicedtyperank) rsuper__init__ models.lorar5regular_linear_layerweightr7r8 in_features out_featureslora_linear_layerr%) selfr=r%r6r9r8deprecation_messager5r7rrrr;Ts"      zPatchedLoraProjection.__init__F destinationprefix keep_varscs6|jdur|jj||||dStj||||dS)NrF)rAr= state_dictr:)rBrGrHrIargsrDrrrJos z PatchedLoraProjection.state_dictr$c Cs|jdurdS|jjjj|jjjj}}|jjj}|jjjj}|jjjj}|jj dur;||jj |jj }||t |dddf|dddfd}|rft |rftd|d|j||d|jj_d|_||_||_||_dSNraThis LoRA weight seems to be broken. Encountered NaN values when trying to fuse LoRA weights for .LoRA weights will not be fused.r7r8)rAr=r>datar8r7floatupdownr6r9torchbmmisnananyitemrtocpuw_upw_downr% rBr% safe_fusingr8r7w_origr[r\ fused_weightrrr _fuse_loraws(  0   z PatchedLoraProjection._fuse_lorac Cst|dddurt|dddusdS|jjj}|j|j}}|jj|d}|j |}||j t |dddf|dddfd}|j||d|jj_d|_d|_ dSNr[r\r7rrO) getattrr=r>rPr8r7r[rYrQr\r%rTrUrBr`r8r7r[r\unfused_weightrrr _unfuse_loras  6 z"PatchedLoraProjection._unfuse_loracCs>|jdurd|_|jdur||S|||j||S)Nr$)r%rAr=)rBinputrrrforwards   zPatchedLoraProjection.forward)r2Nr3Nr$F) r __module__ __qualname__r;rJrargri __classcell__rrrDrr(Ss   r(csreZdZdZ    ddedededeedeeej e fd eej f fd d Z d ej d ej fddZZS)r5a A linear layer that is used with LoRA. Parameters: in_features (`int`): Number of input features. out_features (`int`): Number of output features. rank (`int`, `optional`, defaults to 4): The rank of the LoRA layer. network_alpha (`float`, `optional`, defaults to `None`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. See https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning device (`torch.device`, `optional`, defaults to `None`): The device to use for the layer's weights. dtype (`torch.dtype`, `optional`, defaults to `None`): The dtype to use for the layer's weights. r3Nr?r@r9r6r7r8cstd}tdd|tj||d||d|_tj||d||d|_||_||_||_ ||_ tj j |jj d|dtj |jj dS)NzmUse of `LoRALinearLayer` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.r5r4F)biasr7r8r2std)r:r;rrLinearrSrRr6r9r@r?initnormal_r>zeros_)rBr?r@r9r6r7r8rCrDrrr;s  zLoRALinearLayer.__init__ hidden_statesreturncCN|j}|jjj}|||}||}|jdur"||j|j9}||Sr&r8rSr>rYrRr6r9rBru orig_dtyper8down_hidden_statesup_hidden_statesrrrri    zLoRALinearLayer.forward)r3NNN)rrkrl__doc__intrrQrrTr7strr8r;TensorrirmrrrDrr5s(r5cseZdZdZ     ddededed eeeeeffd eeeeeffd eeeeefefd ee ffd d Z de j de j fddZ ZS)LoRAConv2dLayera" A convolutional layer that is used with LoRA. Parameters: in_features (`int`): Number of input features. out_features (`int`): Number of output features. rank (`int`, `optional`, defaults to 4): The rank of the LoRA layer. kernel_size (`int` or `tuple` of two `int`, `optional`, defaults to 1): The kernel size of the convolution. stride (`int` or `tuple` of two `int`, `optional`, defaults to 1): The stride of the convolution. padding (`int` or `tuple` of two `int` or `str`, `optional`, defaults to 0): The padding of the convolution. network_alpha (`float`, `optional`, defaults to `None`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. See https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning r3r2r2rNr?r@r9 kernel_sizestridepaddingr6c std}tdd|tj|||||dd|_tj||dddd|_||_||_tj j |jj d|d tj |jj dS) NzmUse of `LoRAConv2dLayer` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr4F)rrrrnr)rrrnr2ro) r:r;rrConv2drSrRr6r9rrrsr>rt) rBr?r@r9rrrr6rCrDrrr;s  zLoRAConv2dLayer.__init__rurvcCrwr&rxryrrrrir}zLoRAConv2dLayer.forward)r3rrrN)rrkrlr~rrrrrrQr;rTrrirmrrrDrrs.rcszeZdZdZdddeeffddZdeefddZdd ed e fd dZ ddZ dde j dede j fddZZS)LoRACompatibleConvz; A convolutional layer that can be used with LoRA. N lora_layerrc,d}tdd|tj|i|||_dS)NzpUse of `LoRACompatibleConv` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr4rr:r;rrBrrKkwargsrCrDrrr;0  zLoRACompatibleConv.__init__cCd}tdd|||_dSNznUse of `set_lora_layer()` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.set_lora_layerr4rrrBrrCrrrr7s  z!LoRACompatibleConv.set_lora_layerr$Fr%r^c Cs|jdurdS|jjj|jjj}}|jj}|jjjj}|jjjj}|jjdur8||jj|jj }t |j dd|j dd}| |j}|||} |ret | retd|d| j||d|j_d|_||_||_||_dS)Nr2 start_dimrMrNrO)rr>rPr8r7rQrRrSr6r9rTmmflattenreshapeshaperVrWrXrrYrZr[r\ _lora_scale) rBr%r^r8r7r_r[r\fusionr`rrrra=s,        zLoRACompatibleConv._fuse_loracCst|dddurt|dddusdS|jj}|jj|jj}}|jj|d|_|j||_t |jj dd|jj dd}| |j }||j|}|j||d|j_d|_d|_dS)Nr[r\rcr2rrO)rdr>rPr8r7r[rYrQr\rTrrrrr)rBr`r8r7rrfrrrrg_s    zLoRACompatibleConv._unfuse_loraruscalervc Csj|jdkrtj||j|jd}d}n|j}t||j|j|j||j |j }|j dur,|S||| |S)Nzeros)mode)rr) padding_modeFpad _reversed_padding_repeated_twicerconv2dr>rnrdilationgroupsr)rBrurroriginal_outputsrrrriqs  zLoRACompatibleConv.forwardrjr$)rrkrlr~rrr;rrQboolrargrTrrirmrrrDrr+s"$rcs~eZdZdZdddeeffddZdeefddZdd ed e fd dZ ddZ dde j dede j ffdd ZZS)LoRACompatibleLinearz4 A Linear layer that can be used with LoRA. Nrrcr)NzrUse of `LoRACompatibleLinear` is deprecated. Please switch to PEFT backend by installing PEFT: `pip install peft`.rr4rrrDrrr;rzLoRACompatibleLinear.__init__cCrrrrrrrrs  z#LoRACompatibleLinear.set_lora_layerr$Fr%r^c Cs|jdurdS|jjj|jjj}}|jj}|jjjj}|jjjj}|jjdur8||jj|jj }||t |dddf|dddfd}|rct | rctd|d|j||d|j_d|_||_||_||_dSrL)rr>rPr8r7rQrRrSr6r9rTrUrVrWrXrrYrZr[r\rr]rrrras(   0   zLoRACompatibleLinear._fuse_lorac Cst|dddurt|dddusdS|jj}|j|j}}|jj|d}|j|}||j t |dddf|dddfd}|j||d|j_d|_d|_dSrb) rdr>rPr8r7r[rYrQr\rrTrUrerrrrgs 6 z!LoRACompatibleLinear._unfuse_lorarurrvcs8|jdur t|}|St||||}|Sr&)rr:ri)rBruroutrDrrris  zLoRACompatibleLinear.forwardrjr)rrkrlr~rr5r;rrQrrargrTrrirmrrrDrrs (rr) typingrrrrTtorch.nn.functionalr functionalrutilsrrutils.import_utilsr transformersr r get_loggerrloggerrr#rQr1Moduler(r5rrrrqrrrrrs"   \<@W