o Si[@s"ddlZddlZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl m Z ddlmZdZ    dd e ed e ed ed e ed e eegdff ddZ    dded e ed e ed ed e ed df ddZd e eeddfd efddZd efddZd efddZdS)N)partial)CallableLiteralOptionalUnion) distributed)fix_random_seedLHOTSE_PROCESS_SEEDT*rank world_size#set_different_node_and_worker_seedsseedreturncCstt||||dS)a Calling this function creates a worker_init_fn suitable to pass to PyTorch's DataLoader. It helps with two issues: * sets the random seeds differently for each worker and node, which helps with avoiding duplication in randomized data augmentation techniques. * sets environment variables that help WebDataset detect it's inside multi-GPU (DDP) training, so that it correctly de-duplicates the data across nodes. r r r r)rworker_init_fnrrN/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/dataloading.pymake_worker_init_fnsr worker_idcCs|r|d|}|dur|d|7}t|t|tjt<|dur'|dur'dS|dur/|dus9Jd|d|t|tjd<t|tjd<dS)z~ Function created by :func:`~lhotse.dataset.dataloading.make_worker_init_fn`, refer to its documentation for details. dNiz!Both args must be not None: rank=z , world_size=RANK WORLD_SIZE)rstrosenvironr )rr r r r process_seedrrrr)s  r)trng randomizedcCst|tr|S|durtddS|dkr9tjj}|dur)tddStt j vs2Jdtt j tS|dkrBt dSt dt|d |d ) a Resolves the special values of random seed supported in Lhotse. If it's an integer, we'll just return it. If it's "trng", we'll use the ``secrets`` module to generate a random seed using a true RNG (to the extend supported by the OS). If it's "randomized", we'll check whether we're in a dataloading worker of ``torch.utils.data.DataLoader``. If we are, we expect that it was passed the result of :func:`~lhotse.dataset.dataloading.make_worker_init_fn` into its ``worker_init_fn`` argument, in which case we'll return a special seed exclusive to that worker. If we are not in a dataloading worker (or ``num_workers`` was set to ``0``), we'll return Python's ``random`` module global seed. NrrzRequested seed='randomized' for shuffling shards differently on each DataLoader node and worker, but lhotse.dataset.dataloading.worker_init_fn was not called.rlz-Unexpected type or value of seed: type(seed)=z seed=z<. Supported values are: None, int, 'trng', and 'randomized'.) isinstanceintrandomgetstatetorchutilsdataget_worker_infor rrsecrets randbelow ValueErrortype)r worker_inforrr resolve_seedHs"    r-cC4dtjvr ttjdStrtrtSdS)mSource: https://github.com/danpovey/icefall/blob/74bf02bba6016c1eb37858a4e0e8a40f7d302bdb/icefall/dist.py#L56rr)rrr!dist is_availableis_initializedget_world_sizerrrrr3~ r3cCr.)r/rr)rrr!r0r1r2get_rankrrrrr5r4r5)NNTr )rr"r(sys functoolsrtypingrrrrr$rr0 lhotse.utilsrr r!boolrrr-r3r5rrrrsZ     6