o ,i@sddlZddlmZmZmZmZmZmZmZddl m Z ddl Z ddl m Z ddl mZmZmZee j ee j fZgdZddZe dd ed ed ededeede j f ddZe ded dd ed ed ededeede j f ddZedd ededeeddfddZdS)N)UnionIterableListDictTupleOptionalcast) deprecated)Tensor)"_group_tensors_by_device_and_dtype_has_foreach_support_device_has_foreach_support)clip_grad_norm_clip_grad_normclip_grad_value_csfdd}t||S)z This wrapper is needed to avoid a circular import when using @torch.no_grad on the exposed functions clip_grad_norm_ and clip_grad_value_ themselves. cs:t|i|WdS1swYdSN)torchno_grad)argskwargsfuncV/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch/nn/utils/clip_grad.py_no_grad_wrappers  $z"_no_grad.._no_grad_wrapper) functoolsupdate_wrapper)rrrrr_no_grads  r@F parametersmax_norm norm_typeerror_if_nonfiniteforeachreturncst|tjr |g}dd|D}t|}tt|dkr#tdS|djt|g}g}|D]9\\}} \\} } |durEt | |sK|rUt |rU| t | q3|r`t d|jd| fdd| Dq3tjtfd d|D} |rt| | rt d d || d } tj| d d} |D]<\\}} \\} } |durt | |s|rt |rt| | |q|rt d|jd| |}| D]}||qq| S)aGClip the gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). cSg|] }|jdur|jqSrgrad.0prrr 5z#clip_grad_norm_..rgN:foreach=True was passed, but can't use the foreach API on  tensorscsg|] }tj|qSr)rlinalg vector_norm)r)g)r!rrr+Hscsg|]}|qSr)to)r)norm) first_devicerrr+JszThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)max) isinstancerr floatlentensordevicer itemsr r extend _foreach_norm RuntimeErrortyper/r0stack logical_orisnanisinfclamp _foreach_mul_r2mul_)rr r!r"r#grads grouped_gradsnormsr:_ device_grads total_norm clip_coefclip_coef_clampedclip_coef_clamped_devicer1r)r4r!rrsT    "    rz_`torch.nn.utils.clip_grad_norm` is now deprecated in favor of `torch.nn.utils.clip_grad_norm_`.)categorycCst|||||S)zClip the gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. )r)rr r!r"r#rrrrgsr clip_valuecCst|tjr |g}t|}dd|D}t|g}|D]Q\\}}\\}}|dur5tttt||ds;|rSt |rSt ttt|| t ttt||q|r^t d|j d|D] }tt|j| |dq`qdS)aClip the gradients of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSr%rr&r(rrrr+r,z$clip_grad_value_..N)r:r-r.)minr5)r6rr r7r r;r rrr _foreach_clamp_min__foreach_clamp_max_r>r?clamp_)rrQr#rGrHr:rJr'rrrrxs&  r)rFNr)rtypingrrrrrrrtyping_extensionsr rr torch.utils._foreach_utilsr r r _tensor_or_tensors__all__rr7boolr FutureWarningrrrrrrs\$    M  &