o Xεi, @s~ddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlZddlZddlmZddlmZddZd efd d Z  d)d ede ee eje e eejfe eejfffdedefddZd*ddZded efddZddZ  d+ddZ  d,ddZ d ede eeffdd Z!d ed!e"ddfd"d#Z#d-d$ed%ed&ede efd'd(Z$dS).N)Path)AnyCallableDictListTupleUnion)urlparse)Coqpit)loggercCsrtjdkr!ddl}||jd}||d\}}t|jdd}ntjdkr-td}nt d }| |S) Nwin32rz@Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Foldersz Local AppDataF)strictdarwinz~/Library/Application Support/z .local/share) sysplatformwinregOpenKeyHKEY_CURRENT_USER QueryValueExrresolve expanduserhomejoinpath)appnamerkeydir__ansr>/home/ubuntu/.local/lib/python3.10/site-packages/trainer/io.pyget_user_data_dirs   r configcCsdtj|d}i||}tj|ddd}tj||ddWddS1s+wYdS)a5Copy config.json and other model files to training folder and add new fields. Args: config (Coqpit): Coqpit config defining the training run. out_path (str): output path to copy the file. new_fields (dict): new fileds to be added or edited in the config file. z config.jsonwutf8)encoding)indentN)ospathjointo_dictfsspecopenjsondump)r!out_path new_fieldscopy_config_path new_configfrrrcopy_model_files!s  "r4Tr( map_locationcachereturncKstj|p tj|}|r>|s>tjd|dttdidd}tj |fd|i|WdS1s7wYdSt|d}tj |fd|i|WdS1sZwYdS)aLike torch.load but can load from other locations (e.g. s3:// , gs://). Args: path: Any path or url supported by fsspec. map_location: torch.device or str. cache: If True, cache a remote file locally for subsequent calls. It is cached under `get_user_data_dir()/trainer_cache`. Defaults to True. **kwargs: Keyword arguments forwarded to torch.load. Returns: Object stored in path. z filecache:: cache_storage tts_cacherb) filecachemoder5N) r'r(isdirisfiler+r,strr torchload)r(r5r6kwargsis_localr3rrr load_fsspec3s$$rDFcCs@t|tdd}||d|r||r|||fS)Ncpu)r5model)rDr@deviceload_state_dictcudaeval)rFcheckpoint_pathuse_cudarJstaterrrload_checkpointOsrNrMcKsFt|d}tj||fi|WddS1swYdS)zLike torch.save but can save to other locations (e.g. s3:// , gs://). Args: state: State object to save path: Any path or url supported by fsspec. **kwargs: Keyword arguments forwarded to torch.save. wbN)r+r,r@save)rMr(rBr3rrr save_fsspecYs"rQc Kst|dr |j} n|} t|trdd|D} nt|tr+dd|D} n |dur3|nd} t|trBdd|D} n |durJ|nd} t|trU|}|| | | ||t j dd} | ||rs|| |dSt| |dS) NmodulecSg|]}|qSr state_dict).0optimrrr kzsave_model..cSsi|] \}}||qSrrT)rVkvrrr mszsave_model..cSrSrrT)rVsrrrrXrrYz %B %d, %Y)r!rF optimizerscalerstepepochdate)hasattrrRrU isinstancelistdictitemsr r*datetimerbtodaystrftimeupdaterQ) r!rFr^r_ current_stepra output_path save_funcrB model_stateoptimizer_state scaler_staterMrrr save_modeles2       rrc Ks`d|d} tj|| } td| t||||||| fd|i| |dur.t||dSdS)N checkpoint_.pthz > CHECKPOINT : %srn)r'r(r)r inforrkeep_n_checkpoints) r!rFr^r_rlra output_foldersave_n_checkpointsrnrB file_namerKrrrsave_checkpoints&    rzc Ks:t|tr'|dduo|ddu} | r|d|dkp%| o%|d|dk}n||k}t| ttfr BEST MODEL : %s) model_lossrnzbest_model*.pthzbest_model.pth)rdrfintfloatr'r(r)r rurrr+ get_mapperfsglobbasenamermcopy) current_loss best_lossr!rFr^r_rlrar/ keep_all_best keep_afterrnrB use_eval_loss is_save_modelbest_model_namerKr model_names model_name shortcut_name shortcut_pathrrrsave_best_modelsN        rc szt|j}|tj|d}t|jr'| dr'fdd|D}i}i}dD]Yd}d}|D]#}t d|}|durXt | d} |dusT| |krX| }|}q5fd d|D} |durzt| dkrzt| tjjd }t|d }|dur||<||<q-|std |d d|vr|d|d<nd|vr|d|d<n|d|dkr|d|d<|d|dfS)anGet latest checkpoint or/and best model in path. It is based on globbing for `*.pth` and the RegEx `(checkpoint|best_model)_([0-9]+)`. Args: path: Path to files to be compared. Raises: ValueError: If no checkpoint or best_model files are found. Returns: Path to the last checkpoint Path to best checkpoint z*.pth://csg|]}d|qS)rr)rVry)schemerrrXz'get_last_checkpoint..) checkpoint best_modelN _([0-9]+)rcsg|]}|vr|qSrr)rVfnrrrrX rrr`z!No models found in continue path !rr)r+rrrr'r(r)r r startswithresearchrgroupslenmaxgetctimerD ValueError) r(r file_names last_modelslast_model_numslast_model_num last_modelrymatch model_numkey_file_namesr)rrrget_last_checkpointsF    rncCsHt|j}t|d}t||kr |d| D] }||qdSdS)zKeep only the last n checkpoints in path. Args: path: Path to files to be compared. n: Number of checkpoints to keep. rN)r+rrsort_checkpointsrr)r(rrrryrrrrv#s    rvrmcheckpoint_prefix use_mtimecCsg}ddt||dD}|D]1}|r#|tj||fqtd|d|}|durD|durD|t |d|fqt |}dd|D}|S) a)Sort checkpoint paths based on the checkpoint step number. Args: output_path (str): Path to directory containing checkpoints. checkpoint_prefix (str): Prefix of the checkpoint files. use_mtime (bool): If True, use modification dates to determine checkpoint order. cSsg|]}t|qSr)r?)rVxrrrrX;rYz$sort_checkpoints..z_*z.*rNrcSsg|]}|dqS)r)rVrrrrrXFrY) rrappendr'r(getmtimerrrrsorted)rmrrordering_and_checkpoint_pathglob_checkpointsr( regex_matchcheckpoints_sortedrrrr1sr)NT)FF)NN)FrN)F)%rhr-r'rrpathlibrtypingrrrrrr urllib.parser r+r@coqpitr trainer.loggerr r r4r?rGboolrDrNrQrrrzrrrrvrrrrrsL     ,   , ) :A$