o i @sddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl Z edddgZded ed e efd d Zd e eege egefffd dZde eefded efddZ dde dede e e jd efddZdS)N) namedtuple)fields)AnyCallableDictListOptionalItem constructorconfigroot base_modulereturncCsjg}t|D]+}|dr2|ds2|d|d}|tjvr2d||g}t || |q|S)N).pyz.pyc_r.) oslistdirendswith startswithfindsysmodulesjoin importlib import_moduleappend)r r rfilemodule module_namer B/home/ubuntu/.local/lib/python3.10/site-packages/xformers/utils.pyimport_all_moduless   r"cs"|fdtdtffdd }|S)Nnamer csfdd}|S)zRegisters a subclass. This decorator allows xFormers to instantiate a given subclass from a configuration file, even if the class itself is not part of the xFormers library.cspvr tdt|std|jj|jvr(td|jt|d<|j|S)Nz#Cannot register duplicate item ({})z,Item ({}: {}) must extend the base class: {}z3Cannot register item with duplicate class name ({}))r r ) ValueErrorformat issubclass__name__r add)cls)class_registryr r# name_registryreference_classr r! register_cls+s"    zCget_registry_decorator..register_item..register_clsr )r#r r-r*r+r,)r r#r! register_item$sz-get_registry_decorator..register_item)strr)r*r+r,default_configr/r r.r!get_registry_decorator!sr2superset config_classcsXttddt|fdd|D}D] }||vr$d||<q|di|S)zaGiven a superset of the inputs and a reference config class, return exactly the needed configcSs|jS)N)r#)xr r r!Isz*generate_matching_config..csi|] \}}|vr||qSr r ).0kv field_namesr r! Jsz,generate_matching_config..Nr )listmapritemskeys)r3r4subsetr8r r:r!generate_matching_configDs rBfnrep grad_to_nonec Cstjtjkrtd||dur&|D]}||dd|_qtj}tj | |Wdn1s?wYtj tjj dd}tjj dd}| | | tj ||}tdt||}tj}tj |t|D]} |dur|D]}d|_q|qWdn1swYtj g} d} t| D]+} tjj dd}tjj dd}| | | tj | |||g7} qtt| S)a Benchmark the runtime of the provided function. Args: fn: Function to benchmark rep: Repetition time (in ms) grad_to_none: Reset the gradient of the provided tensor to None Returns: Benchmarked runtime in ms zQCannot capture graph in default stream. Please use side stream in benchmark code.NT) enable_timing )torchcudacurrent_streamdefault_stream RuntimeErrordetach_requires_grad_grad CUDAGraphgraph synchronizeEventrecordreplay elapsed_timemaxintrangemeantensoritem) rDrErFr5g start_event end_event estimate_msn_repeatiret n_retriesr r r!do_bench_cudagraphUsX           rg)rCN)rrr collectionsr dataclassesrtypingrrrrrrJr r0r"r2rBrZTensorfloatrgr r r r!s.    #