o 8wi$4@szdZddlZddlZddlZddlmZmZmZmZddl m Z ddl m Z m Z mZddlmZeeZGdddZdS) zConvenience functions for the simplest parameter transfer cases. Use `speechbrain.utils.checkpoints.Checkpointer` to find a checkpoint and the path to the parameter file. Authors * Aku Rouhe 2020 * Andreas Nautsch 2023 * Adel Moumen 2023 N)DEFAULT_LOAD_HOOKSDEFAULT_TRANSFER_HOOKS PARAMFILE_EXTget_default_hook) run_on_main) FetchSource LocalStrategyfetch) get_loggerc@seZdZdZ     dddZddZddZd d Zd d Zd dZ e ddZ dde j fde fddZddZddZddZdS) PretraineraOrchestrates pretraining First optionally collects files from some source (local directory, HuggingFace repository, base URL), into the `collect_in` directory, if specified. Then, calls load hooks for each of those files. Arguments --------- collect_in : str or Path, optional Path to directory where the files are to be collected. If `None`, then files will be referred to from cache or directly, if possible (URLs will fail). There will not be a centralized target directory with all the files. loadables : mapping Mapping from loadable key to object. This connects the keys to the actual object instances. paths : mapping Mapping from loadable key to filepath. The last part of the path is treated as file name, the rest of it is treated as a "source" which can be either a directory path or a magic source like Huggingface hub ID. e.g. sb/asr-crdnn-libri/lm.ckpt -> source=sb/asr-crdnn-libri, file=lm.ckpt Note that when collecting, you can specify a default source, which is used for all loadables that don't have a path specified. custom_hooks : mapping Mapping from loadable key to parameter transfer hook function. If you want to use a custom loading function, specify it here. conditions: mapping An optional mapping from loadable keys to condition values, useful for loading certain elements only if a flag is turned on NcCsti|_|||dur||i|_|dur||i|_|dur)||i|_|dur5||g|_ dS)N) loadablesset_collect_in add_loadablespaths add_paths custom_hooksadd_custom_hooks conditionsadd_conditionsis_local)self collect_inr rrrra/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/speechbrain/utils/parameter_transfer.py__init__Bs      zPretrainer.__init__cCs"|dur t||_dSd|_dS)zChange the collecting pathN)pathlibPathr)rpathrrrr [s"zPretrainer.set_collect_incC|j|dS)zUpdate the loadables dict from the given mapping. Arguments --------- loadables : mapping Mapping from loadable key to object N)r update)rr rrrr_szPretrainer.add_loadablescCr)aUpdate the paths for different loadables. When collecting parameters, paths here are preferred. Note that when collecting, you can specify a default source, which is used for all loadables that don't have a path specified. Arguments --------- paths : mapping Mapping from loadable key to filepath. The last part of the path is treated as file name, the rest of it is treated as a "source" which can be either a directory path or a magic source like Huggingface hub ID. e.g. sb/asr-crdnn-libri/lm.ckpt -> source=sb/asr-crdnn-libri, file=lm.ckpt N)rr)rrrrrriszPretrainer.add_pathscCr)aLUpdate the custom hooks. When loading parameters, hooks here are preferred over class defaults. Arguments --------- custom_hooks : mapping Mapping from loadable key to parameter transfer hook function. If you want to use a custom loading function, specify it here. N)rr)rrrrrr|s zPretrainer.add_custom_hookscCr)zUpdate the conditions. Arguments --------- conditions: mapping Mapping from loadable keys to condition values, useful for loading certain elements only if a flag is turned on N)rr)rrrrrrs zPretrainer.add_conditionscCs<dd}t|tr|\}}||\}}t|||fS||S)a'Splits a path to source and filename This also handles URLs and Huggingface hub paths, in addition to regular paths. Arguments --------- path : str Returns ------- str Source str Filename cSsd|vr |jdddSd|fS)zCore function to split path./)maxsplitz./)rsplit)srcrrrsplitsz$Pretrainer.split_path..split) isinstancer)rr% fetch_from fetch_pathsourcefilenamerrr split_paths   zPretrainer.split_pathFlocal_strategyc sD|jdur'td|jd|jjddtdkr&|tjkr&t dntdi}|j D]n}| |s9q1|t }||j vrM||j |\}}n|durV|}|}ntd |d |||jd ||d|d } dfd d} t| | | | d||<t|tr|\} }td|dt|j |<|j|q1|S)aRFetches parameters from known paths with fallback default_source The actual parameter files may reside elsewhere, but this ensures a symlink in the self.collect_in directory. The symlink always uses the loadable key in the filename. This standardization makes it easier to orchestrate pretraining on e.g. distributed setups. Use the default_source if you have everything organized neatly into one location, like a Huggingface hub repo. Arguments --------- default_source : str or Path or FetchSource This is used for each loadable which doesn't have a path already specified. e.g. if the loadable has key `"asr"`, then the file to look for is `/asr.ckpt` use_auth_token : bool (default: False) If true Huggingface's auth_token will be used to load private models from the HuggingFace Hub, default is False because the majority of models are public. local_strategy : speechbrain.utils.fetching.LocalStrategy The fetching strategy to use, which controls the behavior of remote file fetching with regards to symlinking and copying. Ignored if a `collect_in` directory was not specified. See :func:`speechbrain.utils.fetching.fetch` for further details. Returns ------- dict Mapping from loadable key to a local path from which loadable's parameters can be loaded. This is not used in this class, but can possibly be helpful. Nz2Collecting files (or symlinks) for pretraining in .T)exist_okWindowszRequested Pretrainer collection using symlinks on Windows. This might not work; see `LocalStrategy` documentation. Consider unsetting `collect_in` in Pretrainer to avoid symlinking altogether.z.run_fetch)r5 post_func post_kwargszSet local path in self.paths["z"] = )rloggerdebugmkdirplatformsystemrSYMLINKwarningswarnr is_loadablerrr+ ValueErrorrr&rstrrappend) rdefault_sourcer3r,loadable_pathsnamer2r)r* fetch_kwargsr7 _fetch_fromrr6r collect_filessf (         zPretrainer.collect_filescCs.||jvrdS|j|}t|r|St|S)a7Returns True if no condition is defined or for the specified loadable or if the condition is true Arguments --------- name: str the name of the loadable Returns ------- is_loadable: bool whether the item should be loaded T)rcallablebool)rrH conditionrrrrB1s  zPretrainer.is_loadablecCstdd|ji}|jD];}||sq|t}||jvr8td|d|j||j|||<q|j durE|j |||<qt d|d| |dS)z)Loads the files that have been collected.zLoading pretrained files for: z, z'Redirecting (loading from local path): z -> Nz Pretrainer has never collected `z`, did you forget a call to `collect_files`? Could not fall back to `collect_in`, as it was not specified (default is no longer "model_checkpoints").) r:infojoinr rBrrr;rrrC_call_load_hooks)r paramfilesrHr*rrrload_collectedGs&     zPretrainer.load_collectedcCs|jD]H\}}||sq||}||jvr!|j|||qt|t}|dur0|||qt|t}|durBd}||||qdt|d}t|dS)NFzDon't know how to load zO. Register default hook or add custom hook for this object.) r itemsrBrrrrtype RuntimeError)rrRrHobjloadpath default_hook end_of_epochMSGrrrrQ_s&      zPretrainer._call_load_hooks)NNNNN)__name__ __module__ __qualname____doc__rr rrrr staticmethodr+rr?rKrBrSrQrrrrr s0&   # y r )r_rr=r@speechbrain.utils.checkpointsrrrrspeechbrain.utils.distributedrspeechbrain.utils.fetchingrrr speechbrain.utils.loggerr r\r:r rrrrs