o wi#@sddlmZmZddlmZddlZddlZddlmZddl m Z GdddeZ Gdd d e Z Gd d d eZ Gd d d e ZdS))ABCabstractmethod)TupleN)mask_sequence_tensor)loggingc @s"eZdZdZddedefddZedejd ejd ejd ejfd d Z edejd ejd ejd ejfddZ edejd ejd ejdejd ejf ddZ e dejde d ejfddZd de dejjd ejfddZdejd ejd ejd ejfddZdejd ejd ejdejd ejf ddZdS)!ConditionalFlowa- Abstract class for different conditional flow-matching (CFM) classes Time horizon is [time_min, time_max (should be 1)] every path is "conditioned" on endpoints of the path endpoints are just our paired data samples subclasses need to implement mean, std, and vector_field :0yE>?time_mintime_maxcCs||_||_dSNr r )selfr r ri/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/audio/parts/submodules/flow.py__init__$s zConditionalFlow.__init__timex_startx_endreturncCdS)zF Return the mean of p_t(x | x_start, x_end) at time t Nrrrrrrrrmean(zConditionalFlow.meancCr)zT Return the standard deviation of p_t(x | x_start, x_end) at time t Nrrrrrstd/rzConditionalFlow.stdpointcCr)zS Compute the conditional vector field v_t( point | x_start, x_end) Nr)rrrrrrrr vector_field6szConditionalFlow.vector_fieldn_dimcCs:|jdkrddgdg|d}t|d|}|S)zK Broadcast time tensor to the desired number of dimensions  B1zB -> )ndimjoineinops rearrange)rr target_shaperrr_broadcast_time?s zConditionalFlow._broadcast_timeN batch_sizerngcCs"tj|f|d|j|j|jS)z Randomly sample a batchsize of time_steps from U[self.time_min, self.time_max] Supports an external random number generator for better reproducibility ) generator)torchrandr r )rr(r)rrr generate_timeJs"zConditionalFlow.generate_timecCsB|j||jd}|j|||d}|j|||d}||t|S)z Generate a sample from p_t(x | x_start, x_end) at time t. Note that this implementation assumes all path marginals are normally distributed. rrrr)r'r"rrr+ randn_like)rrrrrrrrrsampleQszConditionalFlow.samplecCs0|j|||d}|j|||d}||||S)zn Compute the conditional flow phi_t( point | x_start, x_end). This is an affine flow. r/)rr)rrrrrrrrrrflow\szConditionalFlow.flow)rr r )__name__ __module__ __qualname____doc__floatrrr+Tensorrrr staticmethodintr'random Generatorr-r1r2rrrrrsD $$  " rc seZdZdZ ddedededeffd d Zd ejd ejd ejdejfddZd ejd ejd ejdejfddZ ddd ejd ejd ejdejdedejf ddZ Z S)OptimalTransportFlowaThe OT-CFM model from [Lipman et at, 2023] Every conditional path the following holds: p_0 = N(x_start, sigma_start) p_1 = N(x_end, sigma_end), mean(x, t) = (time_max - t) * x_start + t * x_end (linear interpolation between x_start and x_end) std(x, t) = (time_max - t) * sigma_start + t * sigma_end Every conditional path is optimal transport map from p_0(x_start, x_end) to p_1(x_start, x_end) Marginal path is not guaranteed to be an optimal transport map from p_0 to p_1 To get the OT-CFM model from [Lipman et at, 2023] just pass zeroes for x_start To get the I-CFM model, set sigma_min=sigma_max To get the rectified flow model, set sigma_min=sigma_max=0 Args: time_min: minimum time value used in the process time_max: maximum time value used in the process sigma_start: the standard deviation of the initial distribution sigma_end: the standard deviation of the target distribution rr -C6?r r sigma_start sigma_endcshtj||d||_||_td|jjtd|jtd|j td|jtd|jdS)Nr Initialized %s with time_min: %s time_max: %sz sgima_start: %sz sigma_end: %s) superrr?r@rdebug __class__r3r r )rr r r?r@rFrrrszOptimalTransportFlow.__init__rrrrcCs|j||||Sr )r rrrrrrrrszOptimalTransportFlow.meancCs|j||j||jSr )r r?r@rHrrrrszOptimalTransportFlow.stdgư>)epsrrIcCsd|j||jd}|j|jkr||S|j|||j||}d||j||j}|||S)Nr.r)r'r"r?r@)rrrrrrInumdenomrrrrs   z!OptimalTransportFlow.vector_field)rr r r>) r3r4r5r6r7rr+r8rrr __classcell__rrrGrr=hs8" " r=c @sreZdZdZ   ddejjdededefd d Z e d d Z e d ej dej dej deej ej ffddZdS)ConditionalFlowMatchingSampleraH Abstract class for different sampler to solve the ODE in CFM Args: estimator: the NN-based conditional vector field estimator num_steps: How many time steps to iterate in the process time_min: minimum time value used in the process time_max: maximum time value used in the process rr estimator num_stepsr r cCs||_||_||_||_dSr rOrPr r rrOrPr r rrrrs z'ConditionalFlowMatchingSampler.__init__cCs|j|j|jSr )r r rP)rrrr time_stepsz(ConditionalFlowMatchingSampler.time_stepstateestimator_condition state_lengthrcCsdSr r)rrTrUrVrrrforwardsz&ConditionalFlowMatchingSampler.forwardNrNrr )r3r4r5r6r+nnModuler:r7rpropertyrSrr8rrWrrrrrMs4  rMc szeZdZdZ   ddejjdededeffd d Z d d Z e d ej dej dej de ej ej ffddZZS)#ConditionalFlowMatchingEulerSamplerzM The Euler Sampler for solving the ODE in CFM on a uniform time grid rNrr rOrPr r csRtj||||dtd|jjtd|jtd|jtd|jdS)NrQrAz num_steps: %srBrC) rDrrrErFr3rPr r rRrGrrrsz,ConditionalFlowMatchingEulerSampler.__init__cOs|j|i|Sr )rW)rargskwargsrrr__call__sz,ConditionalFlowMatchingEulerSampler.__call__rTrUrVrc Cst|j|j|jd}|durt||}|ddD]9}|tj|jd|jd}|dur1|}n tj ||gdd}|j |||d\}} |||j }|durTt||}q||fS)Nrr)device)dim)input input_length condition) r+linspacer r rPronesshaperacatrOrS) rrTrUrV time_stepstrestimator_inputr_rrrrWs  z+ConditionalFlowMatchingEulerSampler.forwardrX)r3r4r5r6r+rYrZr:r7rr_inference_moder8rrWrLrrrGrr\s2r\)abcrrtypingrr$r+#nemo.collections.common.parts.utilsr nemo.utilsrrr=rMr\rrrrs   P@#