o 闦i @sNddlZddlZddlmZmZmZmZddlmZddl Z ddl m Z m Z ddl mZmZmZgdZdedeefd d Zded eedeefd d ZdedefddZe jfdedededefddZdedefddZ d#dedeedefddZededdededefddZdedefdd Zdede jdefd!d"ZdS)$N)AnyListOptionalSet) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturncCs t|tSN)_get_all_graph_pipes_helperset)rr]/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/torch/utils/data/graph_settings.pyrs rid_cachecCsLg}|D]\}\}}||vrq|||||t||q|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphrrrrs  rrcCs t|tpt|dot|jS)Nr ) isinstancerhasattrinspectismethodr rrrr_is_sharding_datapipe)s   r&num_of_instances instance_idcs(t|}dfdd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. Ncs|D]A\}}d}t|r:|durtd|d|t|j}t|jdkr0|n|jd|}|dur@|}||qdS)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr& RuntimeErrorr# signaturer len parameters)r prev_applieddpr appliedsig_helperr(r'r*rrr5=s,  zapply_sharding.._helperr)r )rr'r(r*rrr4rr 0s r cCs,t|dot|dot|jot|jS)N set_shuffleset_seed)r"r#r$r6r7r%rrr_is_shuffle_datapipeXs   r8shufflecCsb|dur|St|}t|}dd|D}|s%|r%td|}|g}|D]}||q'|S)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) NcSsg|]}t|r|qSr)r8).0piperrr ssz*apply_shuffle_settings..z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)r rwarningswarnr9r6)rr9r all_pipes shufflersshufflerrrrras  rz`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngcCs t||Sr)r )rrCrrrrs rcCst|do t|jS)Nr7)r"r#r$r7r%rrr_is_random_datapipesrDcCst|}t|}t}g}|D]}t||vrqt|r(|||t|q|D]}ttj dtj dj |d }| |q+|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds r)dtype) generator)r rridrDrrinttorchemptyint64random_itemr7)rrCrr?cacherandom_datapipesr; random_seedrrrr s"    r r) r#r=typingrrrrtyping_extensionsrrI(torch.utils.data.datapipes.iter.shardingrrtorch.utils.data.graphr r r __all__rrHrboolr&DEFAULTr r8r FutureWarningrrD Generatorr rrrrsV      (  #