o ꁱi$@sddlZddlmZmZmZddlZddlmZddlmm Z ddl m Z ddl mZddlmZddlmZddlmZeeZGd d d ejZd d ZGd ddejZGdddejZGdddejZGdddejZGdddeZe eedgZ!dS)N)OptionalTupleUnion) AutoModel)PreTrainedModel)ACT2FN)logging)VibeVoiceDiffusionHeadConfigcsFeZdZddedeffdd Zdd Zd d Zd efd dZ Z S)RMSNormư>TFdimepscsHt||_||_||_|jrtt||_ dS| dddS)Nweight) super__init__r relementwise_affinenn Parametertorchonesrregister_parameter)selfr rrmemory_efficient __class__V/home/ubuntu/vibevoice-community/vibevoice/modular/modular_vibevoice_diffusion_head.pyrs zRMSNorm.__init__cCs$|t|djddd|jS)NT)keepdim)rrsqrtpowmeanr)rxrrr_norms$z RMSNorm._normcCs,|||}|jdur||j}|SN)r%floattype_asr)rr$outputrrrforward"s  zRMSNorm.forwardreturncCsd|jd|jd|jS)Nzdim=z, eps=z, elementwise_affine=)r rr)rrrr extra_repr(szRMSNorm.extra_repr)r TF) __name__ __module__ __qualname__intr'rr%r*strr, __classcell__rrrrr s  r cCs|d||S)z!Apply modulation to input tensor.r r)r$shiftscalerrrmodulate+sr5cs8eZdZdZd fdd Zed ddZdd ZZS) TimestepEmbedderz Embeds scalar timesteps into vector representations. Args: hidden_size (`int`): Size of the output embedding frequency_embedding_size (`int`, optional): Size of the intermediate frequency embedding c s@tttj||ddtdtj||dd|_||_dSNFbiassilu)rrr SequentialLinearrmlpfrequency_embedding_size)r hidden_sizer?rrrr8s  zTimestepEmbedder.__init__'cCs|d}tt| tjd|tjd||j}|dddf|d}tj t |t |gdd}|drQtj |t |ddddfgdd}||j S)a Create sinusoidal timestep embeddings. Args: t (`torch.Tensor`): A 1-D Tensor of N indices, one per batch element. These may be fractional. dim (`int`): The dimension of the output. max_period (`int`, optional): Controls the minimum frequency of the embeddings. Returns: `torch.Tensor`: An [N, D] Tensor of positional embeddings. rr)startenddtypeNrr r )rexpmathlogarangefloat32todevicer'catcossin zeros_likerD)tr max_periodhalffreqsargs embeddingrrrtimestep_embeddingBs ( z#TimestepEmbedder.timestep_embeddingcCs|||j}||}|Sr&)rWr?r>)rrQt_freqt_embrrrr*Zs zTimestepEmbedder.forward)r7)rA) r-r.r/__doc__r staticmethodrWr*r2rrrrr60s  r6cs(eZdZdZfddZddZZS)FeedForwardNetworkz Standard feed-forward network with SwiGLU activation. Args: embed_dim (`int`): Input dimension ffn_dim (`int`): Hidden dimension csZt||_tj|j|dd|_tj|j|dd|_tj||jdd|_td|_ dSr8) rr embed_dimrr= gate_projup_proj down_projract_fn)rr]ffn_dimrrrrhs zFeedForwardNetwork.__init__cCs,||}||}||}|||Sr&)r^r_rar`)rr$gateuprrrr*ts   zFeedForwardNetwork.forwardr-r.r/rZrr*r2rrrrr\`s  r\cs,eZdZdZ dfdd ZddZZS) HeadLayera A layer in the diffusion head. Args: embed_dim (`int`): Input dimension ffn_dim (`int`): Hidden dimension cond_dim (`int`): Condition embedding dimension norm_eps (`float`, optional): Epsilon for normalization h㈵>csdt||_||_||_t|j|j|_t|j|d|_t t dt j |d|jdd|_ dS)N)rr;Fr9)rrr]cond_dimrbr\ffnr normrr<rr=adaLN_modulation)rr]rbrinorm_epsrrrrs  zHeadLayer.__init__cCs>||jddd\}}}|||t||||}|S)NrhrrE)rlchunkrjr5rk)rr$c shift_ffn scale_ffngate_ffnrrrr*s zHeadLayer.forwardrgrerrrrrf~s rfcs*eZdZdZdfdd ZddZZS) FinalLayera Final layer in the diffusion head. Args: hidden_size (`int`): Input dimension output_size (`int`): Output dimension cond_size (`int`): Condition embedding dimension norm_eps (`float`, optional): Epsilon for normalization rgcsRtt||dd|_tj||dd|_ttdtj|d|dd|_ dS)NF)rrr9r;r) rrr norm_finalrr=linearr<rrl)rr@ output_size cond_sizermrrrrs  zFinalLayer.__init__cCs8||jddd\}}t||||}||}|S)NrrrE)rlrnr5rurv)rr$ror3r4rrrr*s zFinalLayer.forwardrsrerrrrrts  rtcs@eZdZdZeZdZdZdZfddZ ddZ ddZ Z S) VibeVoiceDiffusionHeadz Diffusion head model for vibevoice. Args: config (`VibeVoiceDiffusionHeadConfig`): Model configuration latent_size (`int`, optional): Size of the latent space. If not provided, uses `config.latent_size`. Tcst_j_j}tj|jdd_tjjjdd_ t j_ t jj tfddtjD_tj|jjd_dS)NFr9cs"g|] }tjjjdqS))r]rbrirm)rfr@ri rms_norm_eps).0_configrbrrr sz3VibeVoiceDiffusionHead.__init__..)r@rwrxrm)rrr~r@ri latent_sizerr=noisy_images_proj cond_projr6 t_embedderr0head_ffn_ratio ModuleListrange head_layerslayersrtrz final_layerinitialize_weights)rr~rrr}rrs$     zVibeVoiceDiffusionHead.__init__cCstjj|jjdjddtjj|jjdjdd|jD] }tj|jdjdqtj|j jdjdtj|j j jddS)z$Initialize the weights of the model.rg{Gz?)stdrrN) rinitnormal_rr>rr constant_rlrrv)rlayerrrrrs  z)VibeVoiceDiffusionHead.initialize_weightscCsL||}||}||}||}|jD]}|||}q|||}|S)ap Forward pass of the prediction head. Args: noisy_images (`torch.Tensor`): Noisy images/latents to denoise timesteps (`torch.Tensor`): Timesteps for diffusion condition (`torch.Tensor`): Conditioning information Returns: `torch.Tensor`: The predicted noise/velocity )rrrrr)r noisy_images timesteps conditionr$rQrorrrrr*s      zVibeVoiceDiffusionHead.forward) r-r.r/rZr config_classsupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparrr*r2rrrrrys $ry)"rGtypingrrrrtorch.nnrtorch.nn.functional functionalFtransformers.models.autortransformers.modeling_utilsrtransformers.activationsrtransformers.utilsrconfiguration_vibevoicer get_loggerr-loggerModuler r5r6r\rfrtryregister__all__rrrrs*       0& \