o }oik7@sddlZddlmZddlmZmZddlmZmZddl m Z m Z m Z m Z ddlmZgdZedd d d Zd ed efddZdeeBdefddZdeeBdededefddZd#deeBdedefddZdedefdd Zdedefd!d"ZdS)$N) lru_cache) LocalTunnel SSHTunnel) DictConfig OmegaConf)add_taskcheck_if_mountedget_mounts_from_configrun_exp)logging) rrr r get_tunneladd_mount_pathcreate_remote_directorycreate_remote_configcheck_remote_mount_directoriesget_unmounted_filepathget_mounted_filepath)maxsizecKstdi|S)z! Establishing ssh tunnel N)r) ssh_tunnelrr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/parts/nemo_run_utils.pyr 'sr mount_source mount_destcCs|durtdd|vrGt|}d}|D]}|d\}}||kr(||kr(dSq|sE|d|d|td|d|ddSdStd) a Add a mount path to the cluster config. Args: mount_source: The source filepath on the local/remote machine. mount_dest: The destination filepath on the remote/local machine. Must be an absolute path. cluster_config: The cluster config dictionary. NCluster config is not provided.mountsF:zAdded mount path: ``zGNo mounts found in cluster config, can only add to existing mount list.) ValueErrorr splitappendr info)rrcluster_configoriginal_mounts added_mount mount_pathsource destinationrrrr /s r directoryr"cCs |durtdt|tr|g}|ddkr9t|dd}|D]}|jd|dd d td |d q dS|dd kr||dd}|durNtdd|vrX|d|d<tdi|d}|D]}|jd|dd d td |dqcdStd|d)a Create a remote directory on the cluster using the cluster config. **Note**: The ssh tunnel config must be provided in the cluster config for remote directory creation. Args: directory: The directory path to be created on the remote cluster. Can be a single directory path or a list of directory paths. cluster_config: The cluster config dictionary. Nrexecutorlocalrjob_dir mkdir -p FThidewarnzCreated directory:  in local filesystem.slurmr:`ssh_tunnel` sub-config is not provided in cluster_config.r, on remote cluster.Unsupported executor: r) r isinstancestrgetrrunr r!r )r(r"tunneldir_pathssh_tunnel_configrrrrTs,    rconfig config_nameconfig_directorycCs|durtd|ds|d}t|tr|g}t|tr$t|}|ddkrpt|dd}|D]9}t j ||}|j d|d d d |j d |d d d |j d t |d|d d d td|dq4dS|ddkr|dd}|durtdd|vr|d|d<tdi|d}|D]9}t j ||}|j d|d d d |j d |d d d |j d t |d|d d d td|dqdStd|d)a Utility to write a remote config file on the cluster using the cluster config. Args: config: The config dictionary to be written to the file. Can be OmegaConf DictConfig or a dictionary. config_name: The name of the config file to be created. config_directory: The directory path where the config file will be created on the remote machine. Can be a single directory path or a list of directory paths to copy the config file to. cluster_config: The cluster config dictionary. Nrz.yamlr)r*rr+r-FTr.ztouch zecho 'z' > zCreated config file: r1r2rr3r,r4r5r)rendswithr6r7dictrcreater8rospathjoinr9to_yamlr r!r )r=r>r?r"r:r;config_filepathr<rrrrs@      "  " rT directoriesexit_on_failurecCs|durtdt|tr|g}|ddkrXtdd}g}|D]}|jd|dddd }d |jvr8||q t|d krT|rVd d |D}d |}t d|dSdS|ddkr|dd}|durmtdd|vrwt |d<t di|d}g}|D]}|jd|dddd }d |jvr||qt|d kr|rdd |D}d |}t d|dSdStd|d)a Check if files and directories at the source location exist for later mounting on the cluster. Args: directories: The directory path to be checked on the local/remote machine. Can be a single directory path or a list. Can be either a file or a directory. cluster_config: The cluster config dictionary. exit_on_failure: If True, will raise an exception if the directories do not exist at the source location. Nrr)r*r+ztest -e z && echo "Directory Exists"Tr.zDirectory ExistsrcSg|]}|dqSz% DOES NOT exist at source destinationr.0locrrr  z2check_remote_mount_directories.. zOSome files or directories do not exist at the source location for mounting !! r2rr3r,cSrJrKrrLrrrrOrPr5r)rr6r7r8rr9stdoutr lenrEFileNotFoundErrorrCgetcwdr )rHr"rIr:missing_source_locationsr(resultr<rrrrs`            rfilepathcCsxd}|dD]}|d\}}||r|}nq|dur)td|d|d|d\}}||t|d}|S)ar Resolve the mounted filepath using the cluster config to merge the mount source path to the filepath. Raises an exception if the mount path is not found for the file path. Args: cluster_config: The cluster config dictionary. filepath: The filepath to be unmounted using the cluster config. Returns: str: unmounted filepath Nrr/Could not find a mount path for the file path ``. Below paths are mounted: r startswithrrSr"rXr%mountrrrrrr"   rcCsxd}|dD]}|d\}}||r|}nq|dur)td|d|d|d\}}||t|d}|S)as Resolve the mounted filepath using the cluster config to merge the mount destination path to the filepath. Raises an exception if the mount path is not found for the file path. Args: cluster_config: The cluster config dictionary. filepath: The filepath to be mounted using the cluster config. Returns: str: mounted filepath NrrrYrZr[r]rrrrCr_r)T)rC functoolsrnemo_run.core.tunnelrr omegaconfrr*nemo.collections.common.parts.skills_utilsrrr r nemo.utilsr __all__r r7r listrArrboolrrrrrrrs   %5ET!