o پi@ @sddlZddlZddlZddlmZddlmZddZdd d Zdd d Zd ddZ ddZ   d!dej de de de de f ddZ  d"dej de de de fddZdS)#N)nn)_calculate_fan_in_and_fan_outcCsdd}||d|ks||d|krtjddd||||}||||}|d|dd|d|||td|||j||d|S) NcSsdt|tddS)N?@)matherfsqrt)xr K/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/weight_init.pynorm_cdf sz _trunc_normal_..norm_cdfzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelr)minmax) warningswarnuniform_erfinv_mul_rradd_clamp_)tensormeanstdabr lur r r _trunc_normal_s  r rrcCs<tt|||||WdS1swYdS)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. NOTE: this impl is similar to the PyTorch trunc_normal_, the bounds [a, b] are applied while sampling the normal with mean/std applied, therefore a, b args should be adjusted to match the range of mean, std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) N)torchno_gradr rrrrrr r r trunc_normal_+s $r&cCsNtt|dd|||||Wd|S1s wY|S)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) rrN)r#r$r rrr%r r r trunc_normal_tf_Fs  r'fan_innormalc Cst|\}}|dkr |}n|dkr|}n |dkr||d}||}|dkr3t|t|dddS|dkrWt|jt|dWddS1sPwYdS|d krtd |}t|| |WddS1sywYdStd |) Nr(fan_outfan_avgr truncated_normalg۶%?rr)uniformzinvalid distribution ) rr'rrr#r$normal_r ValueError) rscalemode distributionr(r*denomvarianceboundr r r variance_scaling_cs(   " "r8cCst|ddddS)Nr(r,)r3r4)r8)rr r r lecun_normal_|sr9{Gz?headmodulename init_bias head_biasclassifier_namecCst|tjtjtjtjfrE||r#tj|j tj |j |dStjj |j ddt|tjrA|j durCtj |j |dSdSdSt |drP|dSdS)Nr:r- init_weights) isinstancerLinearConv1dConv2dConv3d startswithinitzeros_weight constant_biasr&hasattrrA)r<r=r>r?r@r r r init_weight_vits   rNcCst|tjr@||rtj|jtj|j|dStj |j|jdur>d|vr5tjj |jddntj|jdSdSt|tj r[t |j|jdurYtj|jdSdSt |drf|dSdS)Nmlpgư>r-rA)rBrrCrGrHrIrJrKrLxavier_uniform_r0rEr9rMrA)r<r=r?r@r r r init_weight_jaxs$   &     rQ)r!rr"r)rr(r))r:r!r;)r!r;)r#rrr torch.nn.initrr r&r'r8r9ModulestrfloatrNrQr r r r sF   #