o پiW=[@sNdZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZmZddlmZmZdd lmZmZdd lmZdd lmZGd d d Zddddddddddddddddddddddddddeedddddddddddejedddf+de e e eefefde e ede e e de ed fd!ed"ed#e!d$e ed%e!d&e d'e"d(ed)e!d*e e"d+e e e e fd,e e e e fd-e d.e d/e d0e e d1e d2e d3e e"d4ed5ed6e"d7e e d fd8e e d fd9e e d:e e"d;e eded?ed@edAedBe!dCe!dDej#dEe e"ejfdFe!dGe"dHe ej$j%j&effXdIdJZ'dS)KaNaFlex data loader for dynamic sequence length training. This module provides a specialized data loader for Vision Transformer models that supports: - Dynamic sequence length sampling during training for improved efficiency - Variable patch size training with probabilistic selection - Patch-level random erasing augmentation - Efficient GPU prefetching with normalization Hacked together by / Copyright 2025, Ross Wightman, Hugging Face N)suppress)partial)CallableDictIteratorListOptionalTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) _worker_init adapt_to_chs)NaFlexMapDatasetWrapperNaFlexCollator)PatchRandomErasing)create_transformc@seZdZdZeededdddddf d ejj j d e e d fd e e d fd e dejdeejde dede de ddfddZdee eeejfejffddZde fddZeddZeddZdS)NaFlexPrefetchLoaderz;Data prefetcher for NaFlex format which normalizes patches.cudaNconstr rloadermean.stdchannelsdevice img_dtypere_probre_modere_count re_num_splitsreturnc Cs||_||_|p tj|_t||}t||}dd|f} ||_tjdd|D||jd| |_ tjdd|D||jd| |_ |dkrQt ||| | |d|_ nd|_ |j d ko]tj|_|j d kohtj|_dS) a<Initialize NaFlexPrefetchLoader. Args: loader: DataLoader to prefetch from. mean: Mean values for normalization. std: Standard deviation values for normalization. channels: Number of image channels. device: Device to move tensors to. img_dtype: Data type for image tensors. re_prob: Random erasing probability. re_mode: Random erasing mode. re_count: Maximum number of erasing rectangles. re_num_splits: Number of augmentation splits. r cSg|]}|dqS.0xr'r'K/home/ubuntu/.local/lib/python3.10/site-packages/timm/data/naflex_loader.py Cz1NaFlexPrefetchLoader.__init__..)rdtypecSr$r%r'r(r'r'r+r,Er-r) erase_probmode max_count num_splitsrNrnpu)rrtorchfloat32rrrtensorviewrrrrandom_erasingtyper is_availableis_cudar3is_npu) selfrrrrrrrr r!r"normalization_shaper'r'r+__init__s8     zNaFlexPrefetchLoader.__init__c cs<d}|jrtjj|jd}ttjj|d}n|jr+tjj|jd}ttjj|d}nd}t }|j D]\}}|| D] \}}t |tj r^|dkrO|jnd}||j|jd|d||<q>|j|jdd}|d} | j} | jdkr| \} } } | | | d |j}n/| jd kr| \} } }}}||jksJd |jd || | | d |j}n td | jd||j|j}|jdur|j||d|ddd}|| |d<Wdn1swY|s||fVnd}|dur|jrtjj|jd|n|jrtjj|jd||}|}q2||fVdS)zIterate through the loader with prefetching and normalization. Yields: Tuple of (input_dict, targets) with normalized patches. T)r)streamNpatches)r non_blockingr.)rrBrz Expected z channels, got z&Unexpected patches tensor dimensions: z. Expected 3 or 5. patch_coord patch_valid)rErFF)r;r4rStreamrrr@r<r3rritems isinstanceTensorrtoshapendimr7r ValueErrorsubrdivrr8getcurrent_stream wait_stream)r=firstr@stream_contextnext_input_dict next_targetkvr.patches_tensororiginal_shape batch_size num_patches patch_pixelsrApatch_hpatch_wr input_dicttargetr'r'r+__iter__Vsh        ,  zNaFlexPrefetchLoader.__iter__cCs t|jS)zhGet length of underlying loader. Returns: Number of batches in the loader. )lenrr=r'r'r+__len__s zNaFlexPrefetchLoader.__len__cC|jjS)zrGet sampler from underlying loader. Returns: Sampler from the underlying DataLoader. )rsamplerrer'r'r+rhzNaFlexPrefetchLoader.samplercCrg)zrGet dataset from underlying loader. Returns: Dataset from the underlying DataLoader. )rdatasetrer'r'r+rjrizNaFlexPrefetchLoader.dataset)__name__ __module__ __qualname____doc__r r r4rutilsdata DataLoaderr floatintrr.strr?rrrJrcrfpropertyrhrjr'r'r'r+rsP      &8N r)@iirx Frrg?g?bilinear*Trall patch_sizepatch_size_choicespatch_size_choice_probstrain_seq_lens. max_seq_lenr\ is_trainingmixup_fnno_augrr r!re_splittrain_crop_modescaleratiohflipvflip color_jittercolor_jitter_probgrayscale_probgaussian_blur_prob auto_augmentnum_aug_repeatsnum_aug_splits interpolationrrcrop_pct crop_modecrop_border_pixels num_workers distributedrank world_sizeseedepochuse_prefetcher pin_memoryrrpersistent_workersworker_seedingr#c,4Cs|r|dks Jdttfiddd| d|d|d|d |d |d |d |d |d|d|d|d|d|d|d|d|d| d| d| d|&dd},t|}-||-}.t|tjjjriJdt||,|||||.||$|!|"|#d|%d}/tjjj |/dd| d|'tt |+d|*d }0|&rt |0|||(|)| | | d!}0|0Std||||&d||dd" |_ t |d#}1d}2|!rt|tjjjsdd$lm}3|3|}2tjjj ||d| |2|1|'dd%}0|&rt |0|||(|)d&}0|0S)'aD Create a data loader with dynamic sequence length sampling for training. Args: dataset: Dataset to load from. patch_size: Single patch size to use. patch_size_choices: List of patch sizes for variable patch size training. patch_size_choice_probs: Probabilities for each patch size choice. train_seq_lens: Training sequence lengths for dynamic batching. max_seq_len: Fixed sequence length for validation. batch_size: Batch size for validation and max training sequence length. is_training: Whether this is for training (enables dynamic batching). mixup_fn: Optional mixup function. no_aug: Disable augmentation. re_prob: Random erasing probability. re_mode: Random erasing mode. re_count: Maximum number of erasing rectangles. re_split: Random erasing split flag. train_crop_mode: Training crop mode. scale: Scale range for random resize crop. ratio: Aspect ratio range for random resize crop. hflip: Horizontal flip probability. vflip: Vertical flip probability. color_jitter: Color jitter factor. color_jitter_prob: Color jitter probability. grayscale_prob: Grayscale conversion probability. gaussian_blur_prob: Gaussian blur probability. auto_augment: AutoAugment policy. num_aug_repeats: Number of augmentation repeats. num_aug_splits: Number of augmentation splits. interpolation: Interpolation method. mean: Normalization mean values. std: Normalization standard deviation values. crop_pct: Crop percentage for validation. crop_mode: Crop mode. crop_border_pixels: Crop border pixels. num_workers: Number of data loading workers. distributed: Whether using distributed training. rank: Process rank for distributed training. world_size: Total number of processes. seed: Random seed. epoch: Starting epoch. use_prefetcher: Whether to use prefetching. pin_memory: Whether to pin memory. img_dtype: Image data type. device: Device to move tensors to. persistent_workers: Whether to use persistent workers. worker_seeding: Worker seeding mode. Returns: DataLoader or NaFlexPrefetchLoader instance. rz=Augmentation repeats not currently supported in NaFlex loaderrTrrrrrrrrrrrrrrrrrrr r!rnaflexFz IterableDataset Wrapper is a WIP) transform_factoryr~rrseq_lensmax_tokens_per_batchrrrrrshufflerN)r)r\rrrhrworker_init_fnr)rrrrrr r!) rrrrrrr~rpatchify)r)OrderedDistributedSampler)r\rrrh collate_fnr drop_last)rrrr)rrmaxrIr4rorpIterableDatasetrrqrr transformrtimm.data.distributed_samplerr)4rjr~rrrrr\rrrrr r!rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrmax_train_seq_lenrnaflex_datasetrrrhrr'r'r+create_naflex_loadersd       8   r)(rnmath contextlibr functoolsrtypingrrrrrr r r4 constantsr r rrrrrrnaflex_random_erasingrtransforms_factoryrrr5rrsrrboolrtr.rorprqrr'r'r'r+s"  $  '          !#$%&'()*+ ,-./