o wi@s ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddl mZddlmZddlZddlZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lm Z ddl!m"Z"ddl#m$Z$e%e&Z'iZ(e GdddZ)de)dee&*j+ddiZ,de)fddZ-de.dee)fddZ/ddZ0ddZ1e dd!e.de2e.fd"d#Z3d$d%Z4dd'e2e5dBd(e5e.Bde5fd)d*Z6d+d,Z7 -dd.e.d/e5d0e5d1e.d2e8d3e5d4e.fd5d6Z9d7d8Z: -dd.e.d/e5d0e5d1e.d2e8d3e5d4e.fd9d:Z;d;d<Zd?d?eZ=d@dAZ>ddBdCZ?e   ddDe.dEe.dFe.dGe.dBdHe.dBdIe.dBf dJdKZ@dLdMZAdNdOZBGdPdQdQeZCdRe5dSe5ddfdTdUZD ddVe dWe.dXe.dYee.dZeEf d[d\ZFddVe d]e.dWe.dZeEfd^d_ZGdd`e.eBfdadbZHddceIe.dBfdddeZJdfdgZKd2e8fdhdiZL j      k   ddle.dceIe.dBdme8dBfdndoZMedpdqZNdddddddkdddd dde;dddkfdre.e2e.Bdse.e2e.Bdte5e2e5Bdue5dBdve.ejOBdBdweEdxe2e.dye.e2e.BdBdce2e.dBdme8dBdzeEfd{d|ZPdd}d~ZQdS)N)contextmanager) dataclass)datetime) lru_cache)Path)Optional) get_token) StreamWatcher)set_nemorun_home)DockerExecutor)SlurmJobDetailsget_packaging_job_key) SSHTunnel) DictConfig)AppStatec@s*eZdZUdZeed<eed<ddZdS) RepoMetadataz3Metadata for a repo that is used in the experiment.namepathcCs8t|jtr t|j|_|jstd|jddS)NzRepository path `z` does not exist.) isinstancerstrrexists ValueErrorselfrg/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/parts/skills_utils.py __post_init__<s   zRepoMetadata.__post_init__N)__name__ __module__ __qualname____doc__r__annotations__rrrrrrr5s  r nemo_skills)rrmetadatacCs*|jtvrtd|jd|t|j<dS)zRegister an external repo to be packaged with the code in the experiment. Args: metadata (RepoMetadata): Metadata for the external repo. zExternal repo z is already registered.N)rEXTERNAL_REPOSr)r$rrrregister_external_repoLs r&rreturncCs|tvrdSt|S)zGet the path to the registered external repo. Args: name (str): Name of the external repo. Returns: A path to the external repo if it is registered, otherwise None. N)r%)rrrrget_registered_external_repoXs r(cCs>t|dgD]}||ddrdSqtd|d)zEWill check that path_to_check is referenced inside one of the mounts.z/nemo_run/code:/nemo_run/code:r#N The path ''' is not mounted. Check cluster config.)get_mounts_from_config startswithsplitr)cluster_config path_to_checkmountrrrcheck_if_mountedgs r2cCsj|durdSt|D]"}||ddr,|dd|t|dddSq td|d)z;Will return the path on the filesystem before it's mounted.Nr)r#rr*r+)r,r-r.lenr)r/rr1rrrget_unmounted_pathos ,r4Texpnamec sfdd}t|tjr||Sztj| }||WdWS1s'wYWdStytz!tj|}||WdWYS1sNwYWYdStys|rktd|gYYSt d|dww)aWill return the handles of the tasks in the experiment. If ignore_finished=True, will only return handles for the tasks that are not yet finished. Useful for filtering handles to set dependencies on. If ignore_exp_not_exists=True, will not raise an error if the experiment does not exist. TODO: it's still possible that job submission fails if the tasks exist when this function is called, but finish before nemo-run submits a new job (which might take minutes) csFg}|jD]}r||jtjtjtjtjfvr ||j qq|SN) jobsstatus_runnerrRUNNINGPENDING SUBMITTEDUNKNOWNappendhandle)exphandlesjobignore_finishedrr _get_handless   z%get_exp_handles.._get_handlesNzExperiment %s not found!z Experiment z not found!) rrun Experiment from_titleFileNotFoundErrorfrom_idAssertionErrorLOGwarningr)r5rDignore_exp_not_existsrEr@rrCrget_exp_handles{s& ( .   rOcCsrd|vrd}|S|d|p|d}t|dtdd}d|jddd |jdd dd |jd d}|S) a2Get the timeout value for a given partition from the cluster configuration. Args: cluster_config (dict): The cluster configuration dictionary partition (str): The partition name to get timeout for Returns: str: Timeout value in the format 'days:hours:minutes:seconds' timeouts10000:00:00:00 partitionz%H:%M:%Sz00:15:00z00:i02dr)<)rstrptimeseconds)r/rRtimeout time_diffrrr get_timeouts  0rYexcludestrategycCs~|pg}t|tr|}||vr|d7}||vs|S|dkr7ddl}|dd}||vr5|dd}||vs+|Std|d) z$Will return a free port on the host.r#randomrNiz Strategy z not supported.)rintr]randintr)r[r\portr]rrr get_free_ports   rbcCsd|d|d|}|S)a2Construct the command for running generation tasks. Args: server_address (str): Address of the server to connect to generation_commands (str): Commands to run for generation Returns: str: Complete command string including environment setup and server connection check znexport PYTHONPATH=$PYTHONPATH:/nemo_run/code && cd /nemo_run/code && echo 'Waiting for the server to start at z' && while [ $(curl -X PUT z8 >/dev/null 2>&1; echo $?) -ne 0 ]; do sleep 3; done && r)server_addressgeneration_commandscmdrrrget_generation_commands  rf server_typenum_gpus num_nodes model_pathr/ server_port server_argsc Cs|}|dkr t||n|dkr|drt|||dkrLtd|gd}d|d|d|d |d |d |d |d |d|d} |ddkrKd}n&|dkrj|dkrXtdd|d|d|d |d } d}ntd|dd| d} | |fS)aConstruct the command for starting a reward model server. Args: server_type (str): Type of server to start ('nemo', 'vllm', etc.) num_gpus (int): Number of GPUs to use num_nodes (int): Number of nodes to use model_path (str): Path to the model cluster_config (dict): Cluster configuration server_port (int): Port for the server server_args (str, optional): Additional arguments for the server Returns: tuple[str, int]: Server command string and number of tasks to run Raises: ValueError: If server type is not supported or configuration is invalid vllm/nemor])r\r[z[python -m nemo_skills.inference.server.serve_nemo_aligner_reward_model ++rm_model_file= trainer.devices= trainer.num_nodes=z' +model.tensor_model_parallel_size=z) +model.pipeline_model_parallel_size=z inference.port= zU & python -m nemo_skills.inference.server.serve_nemo_reward_model inference_port=z& triton_server_address=localhost: executorlocalr#z1VLLM server does not support multi-node execution?python3 -m nemo_skills.inference.server.serve_vllm --model  --num_gpus --port z Server type 'z!' not supported for reward model.Snvidia-smi && cd /nemo_run/code && export PYTHONPATH=$PYTHONPATH:/nemo_run/code && )r2r-rbr) rhrirjrkr/rlrm num_tasksnemo_aligner_reward_model_portserver_start_cmd server_cmdrrrget_reward_server_commandsd     rcCs d}d|d|d|d}|S)zDGenerate Ray server command with head and worker node configuration.z--node-manager-port=12345 --object-manager-port=12346 --dashboard-port=8265 --dashboard-agent-grpc-port=12347 --runtime-env-agent-port=12349 --metrics-export-port=12350 --min-worker-port=14349 --max-worker-port=18349 zif [ "${SLURM_PROCID:-0}" = 0 ]; then echo 'Starting head node' && export RAY_raylet_start_wait_time_s=120 && ray start --head --port=6379 z && z ;else echo 'Starting worker node' && export RAY_raylet_start_wait_time_s=120 && echo "Connecting to head node at $SLURM_MASTER_NODE" && ray start --block --address=$SLURM_MASTER_NODE:6379 z ;fir) start_cmdports ray_start_cmdrrrget_ray_server_cmd=s rc Cs,|}|dkr t||n|dkr|drt|||dkr?d|d|d|d|d|d |d |d }|d d kr>d}nM|dkrYd|d|d|d |d } t| }d}n3|dkr~|dkrdd} nd} d|d|d|d|d | d |d }d}nd|d|d |d }|}d|d } | |fS)z3Generate command to start a model inference server.rnrorpzEpython -m nemo_skills.inference.server.serve_nemo gpt_model_file=rqrrz tensor_model_parallel_size=z" pipeline_model_parallel_size=z ++port=rsrtrurvr#rwrxrysglangz? --dist_init_addr $SLURM_MASTER_NODE --node_rank $SLURM_PROCID rgzApython3 -m nemo_skills.inference.server.serve_sglang --model z --num_nodes zcFORCE_NCCL_ALL_REDUCE_STRATEGY=1 python -m nemo_skills.inference.server.serve_trt --model_path rz)r2r-r) rhrirjrkr/rlrmr{r}start_vllm_cmdmultinode_argsr~rrrget_server_command`s     rcCsdS)z}Get the command to start the sandbox environment. Returns: str: Command to initialize and start the sandbox z/entrypoint.sh && /start.shrrrrrget_sandox_commandsr)kw_onlyc@seZdZUdZdZeed<dZeed<ede fddZ ede fd d Z ede fd d Z ede fd dZ edefddZdS)CustomJobDetailsz Custom job details class for handling SLURM job logs. Extends SlurmJobDetails to manage separate log files for sbatch and srun processes, with configurable prefixes for each type of log. main srun_prefixrg sbatch_prefixr'cCt|j|jdS)z(Path to sbatch standard output log file. %j_sbatch.logrfolderrrrrrstdoutzCustomJobDetails.stdoutcCr)z&Path to srun standard output log file. %j_srun.logrrrrrrr srun_stdoutrzCustomJobDetails.srun_stdoutcCr)z'Path to sbatch standard error log file.rrrrrrstderrrzCustomJobDetails.stderrcCr)z%Path to srun standard error log file.rrrrrr srun_stderrrzCustomJobDetails.srun_stderrcCs|jsJtj|jdS)z~This term will be used to fetch the logs. The command used to list the files is ls -1 {ls_term} 2> /dev/null z *srun.log)rosrjoinrrrrls_terms zCustomJobDetails.ls_termN)rrrr rrr!rpropertyrrrrrrrrrrrs   rcCs>t|ddd}t|}Wd|S1swY|S)z)Read and parse a YAML configuration file.rtzutf-8)encodingN)openyaml safe_load) config_filefinr/rrr read_configs   rcCs8|durdt|ttfr |S|ptjd}|r"tt||dStd|d rGenerate a unique hash string for an SSH tunnel configuration.r)r)tunnelrrr tunnel_hash7s2rcCs0d|vrtdtjddStdi|dS)zCreate ssh tunnel.rzTNo ssh_tunnel configuration found, assuming we are running from the cluster already.rg)rNr)rLinforF LocalTunnelr)r/rrr get_tunnel<s  rc@seZdZdZddZdS) OutputWatcherz3Class for streaming remote tar/compression process.cCst|ddtjgS)z'Process and display a stream of output. )end)printsysrflush)rstreamrrrsubmitHs  zOutputWatcher.submitN)rrrr rrrrrrEs r transferredtotalc Csl||d}dt|dd}tjd|dd|dd |d d dd |d d dd tjd S)zDisplay SFTP transfer progress.d=r>z File Transfer Progress: [z<50z] z.1fz% (r^zMB/zMB)N)r_rrwriter)rrpercentbarrrrprogress_callbackOs   rr remote_dir local_dirremote_tar_dirverbosecCs|d}tj|\}}|r|n|}|d}tj||}tj||} |d|d} t| j} d} z|jddd} | j d k} Wn t yRd} Ynw| rr|rrd |d |d | d |} |j| t gd| dnd |d|d|} |j| | d} |j j }|j|| |rtnddtd| tj|ddt| d}|j|dWdn1swY|jd|ddt| dS)a Downloads a directory from a remote cluster by creating a tar archive and transferring it. Args: tunnel: SSHTunnel connection remote_dir: Path to the directory on remote server local_dir: Local path to save the downloaded directory remote_tar_dir: Optional directory for temporary tar file creation verbose: Print download progress roz.tar.gzzdu -sb z | cut -f1Fzwhich pvT)warnrzcd z && tar --exclude="*.log" -cf - z | pv -s zB -p -t -e -b -F "Compressing Remote Directory: %b %t %p" | gzip > )watchersptyhidez && tar -czf rt)rNcallbackz Transfer complete: )exist_okzr:gzrzrm )rstriprrr.rrFr_rstripexited Exceptionrsessionclient open_sftprrrmakedirstarfiler extractallremove)rrrrrremote_dir_parentremote_dir_nameremote_tar_filename remote_tar local_tarresult total_sizestreaming_possiblecommandsftptarrrrcluster_downloadZsH     r local_filecCs8|jj}|jt|t||rtnddtddS)a Uploads a file to cluster. TODO: extend to a folder. Args: tunnel: SSHTunnel connection local_file: Path to the local file to upload remote_dir: Cluster path where to save the file verbose: Print upload progress Nrz Transfer complete)rrrputrrr)rrrrrrrrcluster_uploads  rrc Cs|t}z3z!|r t|tjgddddj}t|WWt|Stj y7YWt|dSwt|w)zCheck if the path is a git repo. Args: path: Path to the directory to check. If None, will check the current directory. Returns: Path to the repo if it is a git repo, otherwise None. )gitz rev-parsez--show-toplevelT)capture_outputcheckN) rgetcwdchdir subprocessrFrdecoderrCalledProcessError)r original_path repo_pathrrrget_git_repo_paths&    rextra_package_dirsc Cstdj}|rdd|D}dd|D}ng}g}ttdd }tdd}|r`t|dsFt d t |d | t |d n | t |d | t |j t j|||d }n"t d t |d | t |d | t |j t j||d}i}ttdkr||d<tD]6\}} |dkrq| j}t|rt jt ||d||<qt t|d g} t t|j g} t j| | d||<qt j|ddS|S)z`Will check if we are running from a git repo and use git packager or default packager otherwise.r"cSsg|] }tt|dqS)*)rr.0drrr sz get_packager..cSsg|] }tt|jqSr)rrparentrrrrr s-NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECKrNrzoNot running from NeMo-Skills repo, trying to upload installed package. Make sure there are no extra files in %srzdataset/**/*.jsonl)include_patterninclude_pattern_relative_pathcheck_uncommitted_changeszuNot running from a git repo, trying to upload installed package. Make sure there are no extra files in %s)r  relative_pathr#nemo_run)basepathrT) sub_packagersextract_at_root)r(rboolrgetenvrris_dirloggingrMrr>r rFGitArchivePackagerPatternPackagerr3r%itemsHybridPackager) rnemo_skills_dirinclude_patternsinclude_pattern_relative_pathscheck_uncommited_changesr root_package extra_repos repo_name repo_metarepo_include_pattern"repo_include_pattern_relative_pathrrr get_packagersf       r&cCsi}|dg}|D]J}d|vr7|ddkr|d\}}ntd||||<td|q |tjvrMtd|dtj|||<q td|dgd }d d d i}|d g}||D]a}d|vr|ddkr~|d\}}ntd||||<td|qi|tjvrtd|dtj|||<qi||vr||||<td|dqitd|dqi|S)a Will get the environment variables from the cluster config and the user environment. The following items in the cluster config are supported: - `required_env_vars` - list of required environment variables - `env_vars` - list of optional environment variables WANDB_API_KEY, NVIDIA_API_KEY, OPENAI_API_KEY, and HF_TOKEN are always added if they exist. Args: cluster_config: cluster config dictionary Returns: dict: dictionary of environment required_env_varsrr#z.Invalid required environment variable format: z%Adding required environment variable z from environmentRequired environment variable r) WANDB_API_KEYNVIDIA_API_KEYOPENAI_API_KEYHF_TOKENr,cSs ttSr6)rrrrrrIs z#get_env_variables..env_varsz.Invalid optional environment variable format: z%Adding optional environment variable zOptional environment variable z) not found in user environment; skipping.) rcountr.rrrrrr)r/r.r'env_varvaluealways_optional_env_varsdefault_factoriesoptional_env_varsrrrget_env_variables#s@     r5cCs|dg}tt|D]l}||}d|vrtd|d|d\}}|ddkrI|ddkrI|d d}|tjvrDtd |d tj|}|ddkrm|ddkrm|d d}|tjvrhtd |d tj|}|d|}|||<q |S) a Determines if there are mount paths that are being passed via environment variables. Selects the key in the cluster config called `mounts` which is a list of strings. Each string is in the format of `:` where `env_var` is the name of the environment variable. Args: cluster_config (dict): cluster config dictionary Returns: list: updated list of mounts mountsr)zInvalid mount format: zK. The mount path must be separated by a colon.r{}r#r(zN not found in env variables passed in cluster configs.zP not found in env variables passed in cluster configs.)rranger3rr.rr)r/r6mount_idr1 mount_source mount_targetresolved_mountrrrr,as2           r,rF log_prefix slurm_kwargsc Cs\t|}t|}|p |}| durt| } t| d}|ddkr:|dkr'tdd|d<t||d |d|d |d d id S| sAd |d<nd|d|d<t|D] }d|d|d|<qM| pa|d} d|vrid}n|d| }| durwd| ini}|ddur|d|d<|ddur|d|d<ddddd|d|ddd d!| Dg}|d"d#s|dur| d$||d%d&}t j d=id'|d'd| d(|d)|d*t |d+|d,|d-|d.|d/|d0|d"d#s|ndd1|d2t|d3d |t|||d4|d4|d4d5d6d7d8d9d:| d%|d;| d<||p+iS)>aCreate and configure an executor for running tasks locally or on a cluster. Args: cluster_config (dict): Configuration for the cluster/execution environment container (str): Container image to use num_nodes (int): Number of nodes to use tasks_per_node (int): Number of tasks to run per node gpus_per_node (int): Number of GPUs to use per node job_name (str): Name of the job log_dir (str): Directory for storing logs log_prefix (str, optional): Prefix for log files. Defaults to "main" mounts (list, optional): List of volume mounts partition (str, optional): Slurm partition to use time_min (int, optional): Minimum time allocation in minutes dependencies (list, optional): List of job dependencies extra_package_dirs (tuple[str], optional): Additional directories to package heterogeneous (bool, optional): Whether this is part of a heterogeneous job het_group (int, optional): Heterogeneous group index total_het_groups (int, optional): Total number of heterogeneous groups slurm_kwargs (dict, optional): Additional kwargs for Slurm configuration Returns: DockerExecutor | SlurmExecutor: Configured executor instance N)rrurvr#z4Local executor does not support multi-node execution1PYTHONUNBUFFEREDr entrypointrg) container_imagepackageripc_modevolumesntasks_per_noderinetworkr.additional_kwargsz9$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)SLURM_MASTER_NODEz8$(scontrol show hostnames $SLURM_JOB_NODELIST_HET_GROUP_z | head -n1)SLURM_MASTER_NODE_HET_GROUP_rRrPrQtime_min mail_type mail_userz--no-container-mount-homez --overlapz --mpi=pmixz --wait=10z--ntasks-per-node=z--nodes=z--container-env=,cSsg|]}|qSr)r)rkrrrr sz get_executor..disable_gpus_per_nodeFz--gpus-per-node=dependency_typeafteranyaccountnodesrHrrDcontainer_mountstimeadditional_parametersrE gpus_per_node srun_args job_detailsjob_name_prefix_)job_namerrrwait_time_for_group_jobg{Gz?monitor_group_job_wait_time dependencies heterogeneousr.r)r5r,tupler&rr r:rrkeysr>rF SlurmExecutorrrr4)r/ containerrjtasks_per_noderZr_log_dirr?r6rRrMrcrrd het_grouptotal_het_groupsr@r. config_mountsrEgrouprWrYr[rSrrr get_executors+                 roc csh|dg}|}|D]\}}||d|||d<qz dVW||d<dS||d<w)aTemporarily update environment variables in cluster configuration. A context manager that temporarily adds or updates environment variables in the cluster configuration and restores the original configuration when exiting the context. Args: cluster_config (dict): The cluster configuration to modify updates (dict): Dictionary of environment variable updates where keys are environment variable names and values are their values r.rN)rcopyrr>)r/updatesoriginal_env_varsupdated_env_varskeyr1rrrtemporary_env_update#s  rurerhr{ sandbox_portreuse_code_exp reuse_codetask_dependencies run_afterrdc. Cs&|dur:|ddkr:t|ttjfr|g}g}|D]}t|}t|dkr+td|||qt|dkr9d}nd}|durR|ddkrRd| pN| ddvrRd }| dur[t d d } d}g}| dut || }g}g}| dur|d8i| d |i\}}d | vr|d| d} t d8id |d | d| dd|d| dd| d| d|d|d|ddd|d|d|d|d|}!|ddkr|d krd |d!t |}||||!|||d 7}|rt|tr|g}t|tr|g}t|tr|g}t|t|kst|t|krtd"tt|||D]\}"\}#}$}%|ddkrC|%d krCd |%d!t |#}#t|d#| iV||#|t d8id |d |$d|d|%d|d| d| d|d|d|dt|d kr}d$nd%|"d|d|d|d|d|||Wdn 1swYq%|d 7}| rNd&| i}&| d'g}'|'D]}(d(|(vr|(d)r|(d*d}(|(d+|&d(<qt||&a|tt d8id |d |dd,d|ddkr|djnd dd d|d| d| d-td|d|d|dd,d|d|d|d|d|})||)||Wdn 1sEwY|d 7}|ddkrt|}*t|*tjr|r|plt t|*}|durt|trztj|}Wntyt d.|d/tj!|}Ynwt"d0|j#t$|j%d1}+|+|j&|*j'j(vr|j&|*j'j(|+j)},|D]}-|,|-j*_+qt"d2|+ntd3|j#nt|*tjrt,t|*dt|d kr|j-tj.|dd4|dd1|d5S|r||d_/|j-d6d7|D|d1|d5S)9aWrapper for nemo-run exp.add to help setting up executors and dependencies. Note that there are two parameters that control dependencies. - task_dependencies: list of tasks that this task depends on **within the same experiment** - run_after: a string with experiment name or a list of experiment names that this task should run after. Will schedule dependencies on all tasks inside `run_after` experiments. It needs to already be launched and running. Example of how to set task_dependencies: with run.Experiment(expname) as exp: task1 = add_task(exp, ...) task2 = add_task(exp, ..., task_dependencies=[task1]) You can use `reuse_code_exp` to reuse the code from another experiment (and thus avoid costly packaging/ssh uploading). You can provide either experiment name or the experiment object itself. By default we will reuse the code of the first submitted experiment. If you want to avoid this, set `reuse_code=False`. NrurrzMNo pending or running tasks found for experiment %s, cannot set dependencies.cpurRrgr#r])r\r/rh containersrhrjrirZrirMrcr_rjr?serverrr@rdrkrlrvzmpirun --allow-run-as-root -np z bash -c z8Number of commands, containers and num_tasks must match.NEMO_SKILLS_SANDBOX_PORTrmain_ LISTEN_PORTr. PYTHONPATHz PYTHONPATH= z:/appsandboxr6z$Failed to create experiment from id z, trying to find it by titlez'Trying to reuse code from experiment %sznemo-runzSuccessfully reused code from z2Relevant packaging job not found for experiment %sinline)rurrccSsg|]}tj|dqS)r)rFScript)rrrrrr szadd_task..r)0rrrFrGrOr3rLrMextendrrbrroshlexquoter>r_r enumerateziprurpr-rrVrerrREUSE_CODE_EXPrrJrdebugrHr_titler _idtunnelsrtpackaging_jobsdst_pathrEsymlink_from_remote_dirpopaddrhet_group_indices).r@re task_namer/rhr{rirjrjrRrM with_sandboxrv server_configrwrxryrzrrr@rdrc dep_expname exp_handlesrkrrlcommands executorsr~num_server_tasksserver_containerserver_executorcur_idxcur_cmd cur_container cur_taskssandbox_env_updatescurrent_env_varsoverridesandbox_executorr reuse_key reuse_dirrurrradd_task?s-               $                           rcCs~|ddkr|jdd|durdn|ddS|jd|durdn|dt|}t|tjr;t|}|tvr=|t|<dSdSdS)z|If sequential is not specified, using True locally and False otherwise. If it is specified, it will be used as is. rurvFTN)detach tail_logs sequential)rr)rFrrrrr)r@r/rrssh_hashrrrrun_exps    r)TT)NrZ)rg)NN)NNN)NT)Tr6) rNNNNNFNNN)Rrrrrrr contextlibr dataclassesrr functoolsrpathlibrtypingrrrFrhuggingface_hubrinvoker nemo_run.configr nemo_run.core.execution.dockerr nemo_run.core.execution.slurmr r nemo_run.core.tunnelr omegaconfrtorchx.specs.apir getLoggerrrLrrabsoluterr%r&rr(r2r4listrOrYr_rbrfrrrrrrrrrrrrrrrrrrer&r5r,rorurGrrrrrrsb                ($ W* P * 6     J O>B   !       _