o XεiH@s$ddlZddlZddlZddlmZmZmZddlZddl Z ddl m Z ddl m Z mZddlmZddZdd Zd d Zd d ZddZddZ    d*dddedededededeeeffddZdeded e jjde jjjfd!d"Z  d+d#ed$e d%e!d&e j"j#d'ede jjf d(d)Z$dS),N)DictListTuple)logger)NoamLRStepwiseGradualLR)rank_zero_logger_infocCtjdduS)Napex importlibutil find_specrrI/home/ubuntu/.local/lib/python3.10/site-packages/trainer/trainer_utils.pyis_apex_availablercCr )Nmlflowr rrrris_mlflow_availablerrcCr )Naimr rrrris_aim_availablerrcCr )Nwandbr rrrris_wandb_availablerrcCr )Nclearmlr rrrris_clearml_availablerrcCs<tdt|jrtdtntdt|jr%tdttd|jtn tdttdttjrQtjdkrQtd tj ttd tjttd t ttd t ttd t ttdtjjjttdtjjjttdtjjjttdtjjjjtdS)zPrint training environment.z > Training Environment:z | > Backend: Acceleratez | > Backend: Torchz | > Mixed precision: Truez | > Precision: z | > Mixed precision: Falsez | > Precision: float32rz | > Current device: z | > Num. of GPUs: z | > Num. of CPUs: z | > Num. of Torch Threads: z | > Torch seed: z | > Torch CUDNN: z | > Torch CUDNN deterministic: z | > Torch CUDNN benchmark: z | > Torch TF32 MatMul: N)rruse_acceleratemixed_precision precisiontorchcuda is_available device_countcurrent_deviceos cpu_countget_num_threads initial_seedbackendscudnnenabled deterministic benchmarkmatmul allow_tf32)argsconfigrrrprint_training_env"s&      r0F1r. TrainerArgs cudnn_enablecudnn_benchmarkcudnn_deterministicuse_ddpr-returnc Cstjdtjd<dtjvr|durtjt|d}ntj}|dkr4|s4|js4t d|dt |t |tjd<t j |t|tj||pWtjjjtjj_|patjjjtjj_|pktjjjtjj_|pvtjjjjtjjj_tj} | |fS) aSetup PyTorch environment for training. Args: cudnn_enable (bool): Enable/disable CUDNN. cudnn_benchmark (bool): Enable/disable CUDNN benchmarking. Better to set to False if input sequence length is variable between batches. cudnn_deterministic (bool): Enable/disable CUDNN deterministic mode. use_ddp (bool): DDP flag. True if DDP is enabled, False otherwise. allow_tf32 (bool): Enable/disable TF32. TF32 is only available on Ampere GPUs. torch_seed (int): Seed for torch random number generator. Returns: Tuple[bool, int]: is cuda on or off and number of GPUs in the environment. PCI_BUS_IDCUDA_DEVICE_ORDERCUDA_VISIBLE_DEVICESNz [!] zr active GPUs. Define the target GPU by `CUDA_VISIBLE_DEVICES`. For multi-gpu training use `TTS/bin/distribute.py`.PYTHONHASHSEED)rr empty_cacher#environ set_deviceintr!r RuntimeErrorrandomseedstrnp manual_seedr'r(r*r)r+r,r-r ) r.r3r4r5r6 training_seedr-gpunum_gpususe_cudarrrsetup_torch_training_env?s*         rK lr_schedulerlr_scheduler_params optimizercCsN|durdS|dkrt}n|dkrt}nttjj|}||fi|S)aFFind, initialize and return a Torch scheduler. Args: lr_scheduler (str): Scheduler name. lr_scheduler_params (Dict): Scheduler parameters. optimizer (torch.optim.Optimizer): Optimizer to pass to the scheduler. Returns: torch.optim.lr_scheduler._LRScheduler: Functional scheduler. Nnoamlrstepwisegraduallr)lowerrrgetattrroptimrL)rLrMrN schedulerrrr get_schedulerys   rUoptimizer_nameoptimizer_paramslrmodel parameterscCsR|dkrtd}t|d}nttj|}|dur|}||fd|i|S)aRFind, initialize and return a Torch optimizer. Args: optimizer_name (str): Optimizer name. optimizer_params (dict): Optimizer parameters. lr (float): Initial learning rate. model (torch.nn.Module): Model to pass to the optimizer. Returns: torch.optim.Optimizer: Functional optimizer. radamzTTS.utils.radamRAdamNrX)rQr import_modulerRrrSrZ)rVrWrXrYrZmodulerNrrr get_optimizers    r_)Fr1FN)NN)%r r#rBtypingrrrnumpyrErtrainer.loggerr trainer.torchrrtrainer.utils.distributedrrrrrrr0boolr@rKrDrS OptimizerrL _LRSchedulerrUdictfloatnnModuler_rrrrst  " :