o }oi @sFddlZddlZddlZddlmZmZeeZddl Z ddl Z ddl m Z gdZ e je je je je je je jdZdedeefddZd d Zd ed efd dZded efddZddZd*ddZde jde jfddZGddde jZe j j!d+d!e jd"e jd#e"d e jfd$d%Z#Gd&d'd'e jZ$Gd(d)d)e jZ%dS),N)IterableList)if_exist_compute_softmaxflatten)identityhardtanhreluseluswishsilugeluoutfoldfilescCs:tj|sdS|D]}tj|d|sdSq dS)z Returns true if all given files exist in the given folder Args: outfold: folder path files: list of file names relative to outfold F/T)ospathexists)rrfilerW/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/parts/utils.pyr(s rcCs||sgSd}|D] }|dus||kr|}qg}d}|D]}t||}||||7}qg}|D] }|||q2|S)z,Compute softmax probability over raw logits.Ng)mathexpappend)scores max_scorescore exp_scores total_sumxprobsrrrr7s"  riterreturnccs:|D]}t|tst|ts|Vqt|EdHqdS)zFlatten an iterable which contains values or iterables with values. Args: iter: iterable containing values at the deepest level. Returns: A flat iterable containing values. N) isinstancestrrflatten_iterable)r!itrrrr%Ns  r%list_incCs tt|S)zFlatten a list of (nested lists of) values into a flat list. Args: list_in: list of values, possibly nested Returns: A flat list of values. )listr%)r'rrrr_s rcCs$|j}|jj}t|||fi|_dS)z/Apply mixins to a class instance after creationN) __class____name__type)objmixinbase_cls base_cls_namerrrextend_instanceks   r0 c Cstd||}||}g}|D]>}dtj|} | |kr$||q| |kr0|||q||ks6J|| |||} |d| ||| |qtj||j|jdS)Nzapply rope scaling ...r2dtypedevice) loggerinforpirtorchtensorr7r8) freqs scale_factorlow_freq_factorhigh_freq_factorold_context_lenlow_freq_wavelenhigh_freq_wavelen new_freqsfreqwavelensmoothrrrapply_rope_scalingts    rIr=lengthscCs|j^}}}t|jdkr)tj|||j|jdjdd}|t|dk}||St|jdkrLtj|d||j|jdjdd}|t|dk}||St|jd krptj|dd||j|jdjdd}|t|d k}||St d ) z For tensors containing sequences, zero out out-of-bound elements given lengths of every element in the batch. tensor: tensor of shape (B, L), (B, D, L) or (B, D1, D2, L), lengths: LongTensor of shape (B,) r5r6dimzB -> B 1r2z B -> B 1 1r3z B -> B 1 1 1zCCan only mask tensors of shape B x L, B x D x L and B x D1 x D2 x L) shapelenr<onesr7r8cumsumeinops rearrange ValueError)r=rJ batch_size_ max_lengthsmaskrrrmask_sequence_tensors  "rZcs>eZdZd dedeffdd Zdejdejfd d ZZS) ClampActivation? min_value max_valuecst||_||_dSN)super__init__r^r_)selfr^r_r)rrrbs  zClampActivation.__init__inputr"cCstj||j|jdS)N)minmax)r<clampr^r_)rcrerrrforwardszClampActivation.forward)r\r]) r* __module__ __qualname__floatrbr<Tensorri __classcell__rrrdrr[sr[& .>ralphaepscCsN|j}||d|dd}|||t||d}||}|S)zY equation for snake activation function: x + (alpha + eps)^-1 * sin(alpha * x)^2 rr2rKr5)rOreshape reciprocalr<sinpow)rrprqrOrrrsnakes $ rvc<eZdZdZdeffdd ZdejdejfddZZ S) SnakezT Snake activation function introduced in 'https://arxiv.org/abs/2006.08195' channelscs$tttd|d|_dS)Nr2)rarbnn Parameterr<rQrprcryrdrrrbs zSnake.__init__rr"cCs t||jSr`)rvrp)rcrrrrris z Snake.forward r*rjrk__doc__intrbr<rmrirnrrrdrrxsrxcrw) HalfSnakezo Activation which applies snake to the first half of input elements and leaky relu to the second half. rycs0t|d|_t|j|_tj|_dS)Nr5) rarbsnake_channelsrx snake_actr<rz LeakyReLUlrelur|rdrrrbs   zHalfSnake.__init__rr"cCsZ||ddd|jddf}||dd|jdddf}tj||gdd}|S)Nr2rL)rrrr<cat)rcr snake_out lrelu_outoutrrrris""zHalfSnake.forwardr}rrrdrrsr)r1r2r3r4)ro)&loggingrrtypingrr getLoggerr*r9rSr<torch.nnrz__all__IdentityHardtanhReLUSELUSiLUGELUactivation_registryr$rrr%rr0rIrmrZModuler[jitscriptrlrvrxrrrrrs:     $