o ߗi @sUddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddgZes]ddlZGd d d Zd d Zeejd _eejd _dSddlmZddlmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)e*e+Z,e rzddl-m.Z.Wne/ye,0dYnwGdddej1Z2e2Z3e2e4d<d de5fddZ6GdddZddde5de e7dfde e e5dfdefddZdS)!Nreduce)chain)DictListOptionalTuple TYPE_CHECKINGUnion) is_available)not_noneinit_device_mesh DeviceMeshc@s eZdZdS)_DeviceMeshStubN)__name__ __module__ __qualname__rr[/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/torch/distributed/device_mesh.pyrsrcCsdSNrrrrr_init_device_mesh_stubsrztorch.distributed.device_mesh)Backend) _find_pg_by_ranks_and_tag_get_default_group_get_group_tag get_backendget_process_group_ranksget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup split_group) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingc @s&eZdZd%ddZd&ddZddd eed fd eeed fddfd d Z d'ddde eddfddZ d(ddZ ddde efddZ ededefddZededefddZdddedefddZ d'dedede ejddfdd Zdeeed ffd!d"Zdddededfd#d$ZdS))_MeshEnvreturnNcCs"g|_i|_i|_i|_i|_dSr) mesh_stackchild_to_root_mappingmesh_dim_group_optionsroot_to_flatten_mappingflatten_name_to_root_dimsselfrrr__init__Csz_MeshEnv.__init__rcCs t|jdkr td|jdS)Nrz#No device mesh is currently active!)lenr' RuntimeErrorr,rrrget_current_meshOs z_MeshEnv.get_current_mesh device_meshsubmesh_dim_names. submesh_dimscsNfdd|D}j}g}g}d}t||D]K\} } t| dkrO|j| d|| d|d}|| d||t| d7}||j| jdq|| d||j| dqtt|j } |D]} | | ql|j g| |Rj dg|R} }| D]}tj||dd}||vr|}q||_|j|<|S) Ncs g|] }tfdd|dqS)cs|j|Sr)meshsize)xyr3rr`sz5_MeshEnv.create_sub_mesh...r).0mesh_dimr:rr ^s z,_MeshEnv.create_sub_mesh..rr<r/) start_dimend_dimFmesh_dim_names _init_backend)r6zipr0flattenappendr*_dim_group_infoslistrangendimremovepermutereshaperr device_typer()r-r3r4r5slice_dim_size mesh_tensor slice_dim_idxslice_dim_group_infonum_dims_flattenmesh_dim_indices mesh_dim_namemesh_dims_remained_idxidxpg_ranks_by_dimcur_rankmesh_ndsubmesh res_submeshrr:rcreate_sub_meshTsh        z_MeshEnv.create_sub_meshrVc srt|fddt|jD}|sdfdd|D}|jitgttj|j R}||vrLt |ddd|d|j vr_||j vr_|j |St |j}ttjj}|D]}||qqjjg||Rd |}} |D]} tj| |fd } | | vr| } q|j| <| |j i|<t||j|<| S) Ncsg|] }tj|qSr)r rCindex)r=flattened_mesh_dim_name root_meshrrr?sz0_MeshEnv.create_flatten_mesh.._csg|] }tj|qSr)r rC)r=dimrarrr?s z# already exists for submesh of the . z5The mesh_dim_names of submesh and flattened mesh are z-. Please specify another valid mesh_dim_name.r/rC)_mesh_resources get_root_meshr rCjoinr+ setdefaultrrIkeysr1r*mathprodr6r7rJrKrLrMrNrrrOr(tuple) r-r3rVflatten_dims_in_rootinvalid_dim_namesflattened_mesh_dim_sizeremained_dims_in_rootflatten_dim_in_rootrYrZr[flattened_meshres_flattened_meshrrarcreate_flatten_meshsb         z_MeshEnv.create_flatten_meshcCs|j|d}|s |S|Sr)r(get)r-r3rbrrrrhs z_MeshEnv.get_root_meshcCsD||}|j}|r |r t|dksJd|d}|||SdS)z Returns the index of the mesh dim in the root mesh. The device_mesh passed in needs to be sliced out from the root mesh or submesh of the root mesh. r<z"The submesh can only be a 1D mesh.rN)rhrCr0get_mesh_dim_by_name)r-r3rbchild_mesh_dim_nameschild_mesh_dim_namerrrget_root_mesh_dims  z_MeshEnv.get_root_mesh_dimrOcCs t|Sr)_get_device_handle device_countrOrrrnum_devices_per_host z_MeshEnv.num_devices_per_hostcCstt|Sr)rr%rr~rrr num_hostssz_MeshEnv.num_hostscCsT|jdus t|jdkrtd||jvr"td|dd|jt|j|S)NrzNo `mesh_dim_names` found.zMesh dimension 'z' does not exist.z.Available mesh dimensions are: mesh_dim_names=)rCr0KeyErrorr r_)r-r3rVrrrrxs    z_MeshEnv.get_mesh_dim_by_namerdbackend pg_optionscCs||f|j|<dSr)r))r-rdrrrrr_set_mesh_dim_group_optionssz$_MeshEnv._set_mesh_dim_group_optionsc s|||kr td|j|i|j|}g|j|tfdd|Ds4td|ddd}g}|D]4}||vrN||}|d}||n |j|}||f||krltd|dd |d d |}q:|S) z Validate whether the mesh_dim_names is valid for slicing the given device_mesh. If valid, return dim indexes of the slice mesh in the device mesh. z'Cannot create a submesh from a submesh.c3s|]}|vVqdSrr)r=rVvalid_mesh_dim_namesrr 2s  z0_MeshEnv._get_slice_mesh_dims..zInvalid mesh_dim_names z% specified. Valid mesh_dim_names are .r/z specified. z!Found mesh dim indices to slice: rez.Mesh dim indices should be in ascending order.) rhr1r+rjrCallrrGr_) r-r3rCr+curr_idxslice_mesh_dimsrV mesh_indicesnext_idxrrr_get_slice_mesh_dimssD       z_MeshEnv._get_slice_mesh_dimsc Cs||||}|jd|d|j|}|}g}|D]}t|j||fdd}||vr3|j|gng|_| |q|S)z` Return all the submeshes of a given mesh dimension of the device mesh. r/FrB) rxr6swapdimsrNr7rrrOrHrG) r-r3rVr>rYrZ res_submeshesmesh_1dr\rrr_get_all_submeshesSs&    z_MeshEnv._get_all_submeshesr&Nr&rr)r3rr&r)rrrr.r2rstrrintr^rrvrhr{ staticmethodrrrx C10dBackendOptionsrrrrrrrr%Bsj    P B   4r%rgcudarOcCs tt|dS)a: Get the module corresponding to the device_type which is cuda or cuda-like device. For example, when the device_type is cuda, the module `torch.cuda` is returned. Return None when there is no corresponding module for device_type, otherwise return the corresponding module. N)getattrtorchr~rrrr|rs r|c@seZdZUdZeed<ejed<ee edfed<ddddede ejd fdee edfd e d df d d Z ddZ ddZd7ddZd8ddZd efddZddZded e fddZde ee edffd dfddZd9dee eefd efd d!Zd eefd"d#Ze d9dd$d%e eeefdedee ejd fdee edfd df d&d'Zd9deed efd(d)Zed efd*d+Zed e edffd,d-Z d efd.d/Z!d9dee eefd efd0d1Z"d eeefd2d3Z#d9d4eed dfd5d6Z$dS):ra DeviceMesh represents a mesh of devices, where layout of devices could be represented as a n-d dimension array, and each value of the n-d dimensional array is the global id of the default process group ranks. DeviceMesh could be used to describe the layout of devices across the cluster, and serves as a proxy for communication among the device lists within the cluster. DeviceMesh can be used as a context manager. .. note:: DeviceMesh follows SPMD programming model, which means the same PyTorch Python program is running on all processes/ranks in the cluster. Therefore, users need to make sure the `mesh` array (which describes the layout of devices) should be identical across all ranks. Inconsistent `mesh` will lead to silent hang. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh (ndarray): A multi-dimensional array or an integer tensor describing the layout of devices, where the IDs are global IDs of the default process group. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. A reduction over the first dimension of mesh will reduce across columns (0, 4), .. and (3, 7), a reduction over the second dimension of mesh reduces across rows (0, 1, 2, 3) and (4, 5, 6, 7). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rOr6.rCNTrBr$rDr&cCs ||_t|tjr|jjdkrtd|t|tjr%|jtj dntj |dtj d|_ |r5t |nd|_ t |j |_d|_|dkr|rS||tr`tdkr`t|_|j tk}|ddvsqJ|ddkr~|dnd|_dSdS) Ncpuz!`mesh` must be a CPU tensor, got dtypedevicerxlathreadedr)rr<)rO isinstancerTensorrtype ValueErrordetachtortensorr6rnrCrFtolist_flatten_mesh_list _thread_id_get_or_create_default_group_init_process_groupsr r threading get_identrnonzeror7_coordinate_on_dim)r-rOr6rCrD rank_coordsrrrr.s*  zDeviceMesh.__init__cCst}|stt}|j|kr td|d|jdt|j}|sN|rN|}||krF||dkrFtd|d|d|jd| t |t S) Nz2Mesh should not be bigger than default world size z , but found z ranks!rz8DeviceMesh only support homogeneous hardware, but found z ranks and  z devices!) r rrr6numelr1r|rOr} set_devicerr)r-default_initialized world_size device_handlerrrrrs0  z'DeviceMesh._get_or_create_default_groupc Csg}t}|jjdkr:|jtkr:ttt}tj r,t |dkr,t d|ddn|}| t |||jfnt|jjD]}|jd|d|j|}|tjvr^tj|\}}nd\}}|jrmd|j|nd |} d}t|d d} durt|||| d }|D]8} | } | durt | ||| d }|| vrt||krtd |d| d| t t|| |jfqq@||_dS)Nr<gloozcpu:gloo,cuda:nccl mesh_default)rranks group_descr/)NNmesh_ mesh_dim_bound_device_id) parent_pgr split_ranksr)rrrrzFEach device mesh dimension should get only one process group, but got z in !)rr6rKrrrIrJrrr rr!rGr group_namerrNr7rgr)rCrr#rrr0r1r rH) r-dim_group_infos default_groupr dim_grouprdrYrrrrdim_meshsubgroup_ranksrrrrs           zDeviceMesh._init_process_groupscCstj||Sr)rgr'rGr,rrr __enter__bs zDeviceMesh.__enter__cCstjdSr)rgr'pop)r-exc_type exc_value exc_tracebackrrr__exit__hszDeviceMesh.__exit__cCsJ|jsd|jd|jd}|Sd|jd|jd|jd}|S)Nz DeviceMesh('z', )z, mesh_dim_names=)rCrOr6r)r-device_mesh_reprrrr__repr__ls zDeviceMesh.__repr__cCs:t|dd|_|jst|j|jj|j|j|jf|_|jS)N_hash) rrhashrr6shaperOrCrr,rrr__hash__ts zDeviceMesh.__hash__othercCsbt|tsdSt|t|krdS|j|jko0|jj|jjko0|j|jko0|j|jko0|j|jkS)NFT) rridrr6rrOrCr)r-rrrr__eq__s     zDeviceMesh.__eq__cCs||jstdt|tr|fn|}||jkr|St||}tjj t |||}Wd|S1s7wY|S)aU Slice the current DeviceMesh based on the mesh_dim_names given to create a submesh. The submesh created consists of the dimensions and the communicators indicated by ``mesh_dim_names`` Args: mesh_dim_names (Union[str, Tuple[str]]): the name or the tuple of names of the mesh dimension of the DeviceMesh to create the submesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner in a world size of 8. In the first example: Calling mesh_2d["tp"] on rank 0, 1, 2, 3 returns a 1D submesh of DeviceMesh:([0, 1, 2, 3]). Calling mesh_2d["tp"] on rank 4, 5, 6, 7 returns a 1D submesh of DeviceMesh:([4, 5, 6, 7]). Calling mesh_2d["dp"] on rank 0, 4 returns a 1D submesh of DeviceMesh:([0, 4]). Calling mesh_2d["dp"] on rank 1, 5 returns a 1D submesh of DeviceMesh:([1, 5]). Calling mesh_2d["dp"] on rank 2, 6 returns a 1D submesh of DeviceMesh:([2, 6]). Calling mesh_2d["dp"] on rank 3, 7 returns a 1D submesh of DeviceMesh:([3, 7]). In the second example: Calling mesh_3d["dp", "cp"] on rank 0, 1, 4, 5 returns a 2D submesh of DeviceMesh:([[0, 1], [4, 5]]). Calling mesh_3d["dp", "cp"] on rank 2, 3, 6, 7 returns a 2D submesh of DeviceMesh:([[2, 3], [6, 7]]). Calling mesh_3d["cp", "dp"] on rank 0, 1, 4, 5 returns a 2D submesh of DeviceMesh:([[0, 4], [1, 5]]). Calling mesh_3d["cp", "dp"] on rank 2, 3, 6, 7 returns a 2D submesh of DeviceMesh:([[2, 6], [3, 7]]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize a 2D device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh_2d = init_device_mesh(device_type="cuda", (2,4), mesh_dim_names=("dp", "tp")) >>> tp_mesh = mesh_2d["tp"] >>> dp_mesh = mesh_2d["dp"] >>> >>> # Initialize a 3D mesh. >>> mesh_3d = init_device_mesh(device_type="cuda", (2,2,2), mesh_dim_names=("dp", "pp", "cp")) >>> # The order of the mesh_dim_names provided deteremines the order of dimensions in the submesh. >>> dp_cp_mesh = mesh_3d["dp", "cp"] >>> cp_dp_mesh = mesh_3d["cp", "dp"] z1Cannot slice a DeviceMesh without mesh_dim_names!N) rCr1rrrgrr _subclasses fake_tensorunset_fake_temporarilyr^)r-rCrr\rrr __getitem__s"-   zDeviceMesh.__getitem__r>cCst|ds td|jjdkr|durtd|jjddd|jjdkr6|dur6tt|jd dd St|}tj |d}|r[|| vr[||jd dd }tt|St |t rft||n|}tt|j|dd S) a Returns the single ProcessGroup specified by mesh_dim, or, if mesh_dim is not specified and the DeviceMesh is 1-dimensional, returns the only ProcessGroup in the mesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: A :class:`ProcessGroup` object. rHz*DeviceMesh process groups not initialized!r<NFound the DeviceMesh have dimensionsJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.zmIf you want to get the list of all the ProcessGroups in the DeviceMesh,please use `get_all_groups()` instead.r)hasattrr1r6rKr rrHrgrhr*rwrkrrrx)r-r>rbr*rrrr get_groups4   zDeviceMesh.get_groupcsfddtjjDS)z Returns a list of ProcessGroups for all mesh dimensions. Returns: A list of :class:`ProcessGroup` object. csg|]}|qSr)r)r=ir,rrr?sz-DeviceMesh.get_all_groups..)rJr6rKr,rr,rget_all_groupsszDeviceMesh.get_all_groupsrfgroupcCsFt|trLt|}t|tjr||ks#|dur/t|tjs/||kr/tdt|d|tj|dtj d}t |||dd}t |||j fg|_ |St|}t|dkrZtd |durbtd t|tjrr|jtj dd ntj|dtj d}|jt|krtd |d t|dt |||dd}dd|D|_ |S)aM Constructs a :class:`DeviceMesh` with ``device_type`` from an existing :class:`ProcessGroup`. The constructed device mesh has number of dimensions equal to the number of groups passed. If more than one group is passed, then the ``mesh`` argument is required. Nz Invalid mesh z for ProcessGroup with ranks rrFrBrz.Expects at least one ProcessGroup to be passedz0Must pass mesh if passing multiple ProcessGroups)rrzEExpects mesh with ndim equal to number of ProcessGroups but got mesh z and z ProcessGroupscSs g|] }t|t||jfqSr)rrr)r=rrrrr?Jsz)DeviceMesh.from_group..)rr"rrrrrrrrrrrrHrIr0rrrK)rrOr6rC group_ranksr3groupsrrr from_groups\      zDeviceMesh.from_groupcCs|dur |jS|j|Sr)r6rr7)r-r>rrrr7TszDeviceMesh.sizecCs|jjSr)r6rKr,rrrrKWszDeviceMesh.ndimcCs t|jjSr)rnr6rr,rrrr[rzDeviceMesh.shapecCstS)z: Returns the current global rank. )rr,rrrr_szDeviceMesh.get_rankcCs`|jdkr|durtd|jjdd|durd}t||}t|ts*Jdtt|S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r<Nrrrrz1We expect ProcessGroup before calling `get_rank`!)rKr1r6r rrr"r)r-r>mesh_dim_grouprrrget_local_rankes zDeviceMesh.get_local_rankcCs|jr|jSdS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)rr,rrrget_coordinateszDeviceMesh.get_coordinaterVcCs|jstdt||S)a\ Returns a 1D DeviceMesh by flattening the current DeviceMesh. If no mesh_dim_name is provided, the default is a string concatentaing the mesh_dim_names of the given submesh with each mesh_dim_name separated by "_". For example, if we have a 3D mesh DeviceMesh([[[0, 1], [2, 3]], [[4, 5], [6, 7]]], mesh_dim_names=("dp", "cp", "tp")), calling mesh_3d["dp", "cp"]._flatten() will create a 1D submesh DeviceMesh([0, 1, 2, 3], mesh_dim_names=("dp_cp",)) on rank 0, 1, 2, 3 and a 1D submesh DeviceMesh([4, 5, 6, 7], mesh_dim_names=("dp_cp",)) on rank 4, 5, 6, 7. After the flattened dimension is created, to access the flattened dimesnion in mesh_3d, one can use the existing slicing method to obtain the flattened mesh through calling mesh_3d["dp_cp"]. z3Cannot flatten a DeviceMesh without mesh_dim_names!)rCr1rgrv)r-rVrrr_flattens   zDeviceMesh._flattenrrr)%rrr__doc__r__annotations__rrrrr boolr.rrrrrrobjectrrrr"rrrrrr7propertyrKrrrrrrrrrr{sp '   * t  I.  B *rf mesh_shape.rCr&cCs|dur.tt|t|krtdd|t|t|kr.tddt|dt|d|r=|s=td|d d td tjt|tj d  |}Wdn1s[wYt |||d }|S)a Initializes a `DeviceMesh` based on `device_type`, `mesh_shape`, and `mesh_dim_names` parameters. This creates a DeviceMesh with an n-dimensional array layout, where `n` is the length of `mesh_shape`. If `mesh_dim_names` is provided, each dimension is labeled as `mesh_dim_names[i]`. .. note:: `init_device_mesh` follows SPMD programming model, meaning the same PyTorch Python program runs on all processes/ranks in the cluster. Ensure `mesh_shape` (the dimensions of the nD array describing device layout) is identical across all ranks. Inconsistent `mesh_shape` may lead to hanging. .. note:: If no process group is found, init_device_mesh will initialize distributed process group/groups required for distributed communications behind the scene. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". Passing in a device type with a GPU index, such as "cuda:0", is not allowed. mesh_shape (Tuple[int]): A tuple defining the dimensions of the multi-dimensional array describing the layout of devices. mesh_dim_names (Tuple[str], optional): A tuple of mesh dimension names to assign to each dimension of the multi-dimensional array describing the layout of devices. Its length must match the length of `mesh_shape`. Each string in `mesh_dim_names` must be unique. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) Nz"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):rz0Device type with index is not supported but got rezUIf you maintained a 'torch.device' object, it's recommended to pass in 'device.type'.rr)rOr6rC) r0setr1isalpharrarangerlrmrviewr)rOrrCr6r3rrrr s2)   )r)8loggingrlr functoolsr itertoolsrtypingrrrrr r rtorch.distributedr torch.utils._typing_utilsr __all__sysrrmodulesrr torch._C._distributed_c10drr"torch.distributed.distributed_c10drrrrrrrrr r!r"r# getLoggerrlogger numpy.typingr$ ImportErrorwarninglocalr%rgrrr|rrrrrsf        8  0 7