o Gis@sddlZddlmZddlZddlZddlmZddlmm Z ddl m Z m Z ddlmZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(e)e*Z+d2d3ddZ,d2d3ddZ-d2d3ddZ.dddddej/fde0dej1e0Bde2fdd Z3Gd!d"d"Z4Gd#ddejj5eZ6Gd$d%d%ejj5Z7Gd&d'd'ej5Z8Gd(d)d)ej5Z9Gd*d+d+ejj5Z:eGd,d-d-ej5Z;eGd.d/d/ej5Z query_dimrhdim_headdropoutbiasradded_proj_biasout_biasepsout_dimcontext_pre_onlypre_onlyelementwise_affinecst||_| dur| n|||_||_||_||_| dur"| n||_| |_| |_ | dur3| |n||_ ||_ ||_ t jj|| | d|_t jj|| | d|_t jj||j|d|_t jj||j|d|_t jj||j|d|_|j st jg|_|jt jj|j|j|d|jt j||durt jj|| d|_t jj|| d|_t jj||j|d|_t jj||j|d|_t jj||j|d|_t jj|j||d|_|dur| }|!|dS)N)rrr)r)"superr_head_dim inner_dimr~use_biasrrrrrhrrr@nnRMSNormrirjLinearrrr ModuleListrsappendDropoutrkrlrr r!rt_default_processor_cls set_processor)r^r~rhrrrrrrrrrrr processorr[r+r,r_s> zBriaAttention.__init__r#r$r`rarbc stt|jjjddhfdd|D}t|dkr1t d|d|jj j dfd d |D}|j|||||fi|S) Nip_adapter_masksip_hidden_statescs$g|]\}}|vr|vr|qSr+r+).0k_attn_parametersquiet_attn_parametersr+r, s$z)BriaAttention.forward..rzattention_kwargs z are not expected by z and will be ignored.csi|] \}}|vr||qSr+r+)rrw)rr+r, sz)BriaAttention.forward..) setinspect signaturerru parameterskeysitemslenloggerwarningr[r\)r^r#r$r`rakwargs unused_kwargsr+rr,forwards zBriaAttention.forward) rzr{r|FNTTr}NNFTNrv)r\rwrxrWr_available_processorsr?rHboolr_r@ryr __classcell__r+r+rr,rst     9c@eZdZdedeeffdd ZdejdejfddZZ S) BriaEmbedNDr:axes_dimct||_||_dSrrr_r:rr^r:rrr+r,r_   zBriaEmbedND.__init__idsrbc C|jd}g}g}|}|jjdk}|rtjntj}t|D]"}t|j ||dd|f|j dd|d\} } | | | | qtj |dd |j} tj |dd |j} | | fSNr.mpsT)r:rPrMrQr/rrrHr<typer@float32float64rangerVrr:rrJrp r^rn_axescos_outsin_outr9is_mpsrQirFrIrSrTr+r+r,r(     zBriaEmbedND.forward r\rwrxr?listr_r@ryrrr+r+rr,r rc s:eZdZ d dedededeffdd Zd d ZZS) BriaTimestepsr=' num_channelsflip_sin_to_cosdownscale_freq_shiftscalecs,t||_||_||_||_||_dSr)rr_rrrr time_theta)r^rrrrrrr+r,r_*s  zBriaTimesteps.__init__cCs"t||j|j|j|j|jd}|S)N)rrr max_period)rrrrrr)r^ timestepst_embr+r+r,r4szBriaTimesteps.forward)r=r) r\rwrxr?rrHr_rrr+r+rr,r)s rcs$eZdZfddZddZZS)BriaTimestepProjEmbeddingscs.ttddd|d|_td|d|_dS)NTr)rrrr) in_channelstime_embed_dim)rr_r time_projrtimestep_embedder)r^ embedding_dimrrr+r,r_As z#BriaTimestepProjEmbeddings.__init__cCs ||}||j|d}|S)Nr;)rrrp)r^timestepr;timesteps_proj timesteps_embr+r+r,rIs z"BriaTimestepProjEmbeddings.forward)r\rwrxr_rrr+r+rr,r@s rcr) BriaPosEmbedr:rcrrrrrr+r,r_QrzBriaPosEmbed.__init__rrbc Crrrrr+r+r,rVrzBriaPosEmbed.forwardrr+r+rr,rOrrcseZdZ ddededededef fdd Z dd ejd ejd ejde ejejfd Bde ee fd Bde ejejff ddZ Z S)BriaTransformerBlockrms_normư>r0num_attention_headsattention_head_dimqk_normrc stt||_t||_t|||||ddt|d |_tj |ddd|_ t ||dd|_ tj |ddd|_ t ||dd|_dS)NFT) r~rrrhrrrrrrrrzgelu-approximate)r0dim_out activation_fn)rr_rnorm1 norm1_contextrrWrr LayerNormnorm2r ff norm2_context ff_context)r^r0rrrrrr+r,r_os$    zBriaTransformerBlock.__init__Nr#r$tembraattention_kwargsrbcCs||j||d\}}}} } |j||d\} } } }}|pi}|jd|| |d|}t|dkr3|\}}n t|dkr>|\}}}|d|}||}||}|d| dddf|dddf}||}| d|}||}t|dkr|||}| d|}||}||} | d|dddf| dddf} || }||d|}|j t j kr| dd}||fS) Nemb)r#r$rar rr= r+) rrrr unsqueezerrrrr;r@float16clip)r^r#r$rrarnorm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlpnorm_encoder_hidden_states c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlpattention_outputs attn_outputcontext_attn_outputip_attn_output ff_outputcontext_ff_outputr+r+r,rsD     (   (   zBriaTransformerBlock.forward)rrNN)r\rwrxr?strrHr_r@rytupledictrrrr+r+rr,rms8rcseZdZddedededeffdd Z  dd ejd ejd ejd eejejfdBd e e e fdBdeejejff ddZ Z S)BriaSingleTransformerBlock@r0rr mlp_ratioc sxtt|||_t||_t||j|_tj dd|_ t||j||_ t }t ||||d|ddd|_dS)Ntanh) approximateTr)r~rrhrrrrr)rr_r?mlp_hidden_dimrnormrrproj_mlpGELUact_mlpproj_outrWrr)r^r0rrrrrr+r,r_s"   z#BriaSingleTransformerBlock.__init__Nr#r$rrarrbc Cs|jd}tj||gdd}|}|j||d\}} |||} |p$i}|jd||d|} tj| | gdd}| d} | ||}||}|j tj krU| dd}|ddd|f|dd|df}}||fS) Nr=r/r)r#rar rrr+) rrr@rJrrrrrrr;rr) r^r#r$rrar text_seq_lenresidualrgatemlp_hidden_statesrr+r+r,rs(    *z"BriaSingleTransformerBlock.forward)rr )r\rwrxr?rHr_r@ryr rr rrrr+r+rr,rs" rcseZdZdZdZeddddddd d d gd d d f dedededededededededeeffdd Z e d  d&de j de j de j de j de j d e j d!e j deeefd Bd"ed#ee j eBfd$d%ZZS)'BriaTransformer2DModela The Transformer model introduced in Flux. Based on FluxPipeline with several changes: - no pooled embeddings - We use zero padding for prompts - No guidance embedding since this is not a distilled version Reference: https://blackforestlabs.ai/announcing-black-forest-labs/ Parameters: patch_size (`int`): Patch size to turn the input data into small patches. in_channels (`int`, *optional*, defaults to 16): The number of channels in the input. num_layers (`int`, *optional*, defaults to 18): The number of layers of MMDiT blocks to use. num_single_layers (`int`, *optional*, defaults to 18): The number of layers of single DiT blocks to use. attention_head_dim (`int`, *optional*, defaults to 64): The number of channels in each head. num_attention_heads (`int`, *optional*, defaults to 18): The number of heads to use for multi-head attention. joint_attention_dim (`int`, *optional*): The number of `encoder_hidden_states` dimensions to use. pooled_projection_dim (`int`): Number of dimensions to use when projecting the `pooled_projections`. guidance_embeds (`bool`, defaults to False): Whether to use guidance embeddings. Tr=r{&iNF)8r$r patch_sizer num_layersnum_single_layersrrjoint_attention_dimpooled_projection_dimguidance_embedsaxes_dims_ropec st|_jjjj_t| | d_t j| d_ | r)t jd_ t jjj_tj jjj_t fddtjjD_t fddtjjD_tjjddd _t j j||jd d _d_dS) N)r:r)rr)rc$g|]}tjjjjjdqS)r0rr)rrconfigrrrrr]r+r,r.z3BriaTransformer2DModel.__init__..cr,r-)rrr.rrr/r]r+r,r9r0FrrTr)rr_ out_channelsr.rrrr pos_embedr time_embedguidance_embedrrr(context_embedderr@r x_embedderrrr&transformer_blocksr'single_transformer_blocksrnorm_outrgradient_checkpointing) r^r%rr&r'rrr(r)r*r+ rope_thetarrr]r,r_s,       zBriaTransformer2DModel.__init__rr#r$pooled_projectionsrimg_idstxt_idsguidance return_dictrbc  Cs||}||j}|dur||j}nd}|j||jd} |r,| |j||jd7} ||}t|jdkr<|d}t|jdkrG|d}tj ||fdd} | | }t |j D]>\}}t rr|jrr||||| ||\}}n |||| |d\}}| durt|j t| }tt|}|| ||}qZt |jD]V\}}t r|jr||||| ||\}}n |||| |d\}}| durt|jt| }tt|}|dd|jdddf| |||dd|jdddf<q||| }||}| s|fSt|d S) a_ The [`BriaTransformer2DModel`] forward method. Args: hidden_states (`torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. encoder_hidden_states (`torch.FloatTensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states: (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nrrrr/)r#r$rrar=.)sample)r6rpr;r3r4r5rrrr@rJr2 enumerater7is_grad_enabledr:_gradient_checkpointing_funcr?rBceilr8r9rr)r^r#r$r<rr=r>r?rr@controlnet_block_samplescontrolnet_single_block_samplesrrra index_blockblockinterval_controloutputr+r+r,rHs )             zBriaTransformer2DModel.forward) NNNNNNNTNN)r\rwrx__doc__ _supports_gradient_checkpointingrr?rrr_rr@ry LongTensorrr rr rrrr+r+rr,rs   7   rr)rr)>rtypingrnumpyrBr@torch.nnrtorch.nn.functional functionalrYconfiguration_utilsrrloadersrrutilsrr utils.torch_utilsr attentionr r attention_dispatchr cache_utilsr embeddingsrrrmodeling_outputsrmodeling_utilsr normalizationrrr get_loggerr\rr-r5r7rr?rCrHrVrWModulerrrrrrrrr+r+r+r,sZ             BCOT7