o i/@snddlZddlZddlmZmZddlZddlmZddl m Z ddZ Gdd d Z e Gd d d Z dS) N)ListIterable_str_to_data_type) public_apic Csg}d}|D]3}|d}t|dkr1d|vrtd|d|dkr1|t|kr1td|dt|t|}qt|D] }|dgt|q>t|D];\}}|d}t|dkryd|vrg|d}tt|D] }|d|||<qmqOtt|D] }|||||<qqOt|D]3\}}g} d|vr|d}tdt|D]}|D]} t | } ||| |<| | qq| | q|S) Nr|r*z Type/Format 'z' contains both '|' and '*'zType/Format combinations z" contain illegal dependent lengths) splitlen ValueErrorbuiltinsmaxrangeappend enumeratecopydeepcopyextend) io_variants combinationsmx_poly io_variantio_variant_list_jik new_combscnew_cr!V/home/ubuntu/.local/lib/python3.10/site-packages/tensorrt_bindings/plugin/_autotune.py_gen_onesided_combinationssR              r#c@sFeZdZdddZddZdddZd d Zd d Zd efddZ dS)_TypeFormatCombinationrcCs"dg||_dg||_g|_dSN)typeslayoutstactics)selfnumr!r!r"__init__Vs   z_TypeFormatCombination.__init__cCs ||_dSr%)r&)r)r&r!r!r" set_types[s z _TypeFormatCombination.set_typesNcCs*t|tr ||_dS|gt|j|_dSr%) isinstancerr'r r&)r)r'r!r!r" set_layouts^s  z"_TypeFormatCombination.set_layoutscCstt|jt|jfSr%)hashtupler&r'r)r!r!r"__hash__dsz_TypeFormatCombination.__hash__cCs"t|to|j|jko|j|jkSr%)r-r$r&r')r)otherr!r!r"__eq__gs   z_TypeFormatCombination.__eq__returncCs dt|jdt|jdS)N{z, })strr&r'r1r!r!r"__str__ns z_TypeFormatCombination.__str__)rr%) __name__ __module__ __qualname__r+r,r.r2r4r8r9r!r!r!r"r$Us  r$c @sxeZdZ ddededeefddZddeededed dfd d Zdeed dfd d ZddZ ddZ ddZ dS)AutoTuneCombinationNio_typesr'r(cCs|durSdd|dD|_|durd}dd|dD|_t|jdkr3t|jt|jks3Jt|jt|jkrRt|jdksFJ|jdgt|j|_ng|_g|_g|_||_dS) a Construct a set of supported type/format combinations of a plugin's I/O. Any custom *tactic* s per each such type/format combination can also be advertised. A tactic is simply another way to calculate the output of a plugin for the same type/format combination of the I/O (e.g. if there are multiple kernels available). Args: io_types (str, optional): A string representation of a type combination. Valid format is "type0,type1,...,type#io" where 'type' is of the form "TYPE0[sep]TYPE1[sep]...". TYPE is a valid string representation of a `trt.DataType`. These include "FP32" for trt.float32, "FP16" for trt.float16. The string representation of other data types is the same as their name in the trt.DataType enum. [sep] is a valid separator, which is either '|' or '*'. Only one of these separators can appear in a given `io_types`. (1). '|' indicates a dependent combination: the dependence of the type of one I/O to another I/O. e.g. "FP32|FP16,FP32|FP16" indicates the IO can only be both FP32 or both FP16. (2). '*' indicates an independent combination. e.g. "FP32*FP16,FP32|FP16,FP32|FP16" indicates that the first input is independently either FP32 or FP16 regardless of the rest of the IO. layouts (str, optional): A string representation of a format combination. Valid format is "format0,format1,...,format#io" where 'format' is of the form "FORMAT0[sep]FORMAT1[sep]...". FORMAT is a valid string representation of a `trt.TensorFormat`. These are string versions for the enum values of `trt.TensorFormat`. e.g. "LINEAR" for `trt.TensorFormat.LINEAR`. [sep] is a valid separator, which is either '|' or '*'. The rules are the same as for `io_types`. tactics (Iterable[int], optional): Custom tactics for this type/format combination. Each custom tactic must be a positive integer. Defaults to default tactic (0). .. code-block:: python :linenos: :caption: For a plugin with 3 I/Os, I/O indices 0 and 1 are dependently either FP32/FP16 and index 2 is independently FP32/FP16. @trtp.autotune("my::plugin") def autotune(inp0: trtp.TensorDesc, inp1: trtp.TensorDesc, outputs: Tuple[trtp.TensorDesc]) -> List[trtp.AutoTuneCombination]: # The following would result in the following type combinations: # [FP32, FP32, FP32], [FP16, FP16, FP32], [FP32, FP32, FP16], [FP16, FP16, FP16] return [trtp.AutoTuneCombination("FP32|FP16, FP32|FP16, FP32|FP16", "LINEAR", [1, 2])] .. code-block:: python :linenos: :caption: For a plugin with 2 I/Os, the input/output supports either LINEAR or HWC format for FP32 and LINEAR format for FP16. @trtp.autotune("my::plugin") def autotune(inp0: trtp.TensorDesc, outputs: Tuple[trtp.TensorDesc]) -> List[trtp.AutoTuneCombination]: # Even though (FP16, HWC) is not a valid combination (see next example), TRT should intelligently reject those # and pass the following combinations to the impl function: # [{FP32, FP32}, {LINEAR, LINEAR}], [{FP32, FP32}, {HWC, LINEAR}], [{FP16, FP32}, {LINEAR, LINEAR}] return [trtp.AutoTuneCombination("FP32*FP16, FP32", "LINEAR*HWC, LINEAR", [1, 2])] .. code-block:: python :linenos: :caption: For a plugin with 2 I/Os, the input/output supports either LINEAR or HWC format for FP32 and LINEAR format for FP16 (second method). @trtp.autotune("my::plugin") def autotune(inp0: trtp.TensorDesc, outputs: Tuple[trtp.TensorDesc]) -> List[trtp.AutoTuneCombination]: # We can use two AutoTuneCombination objects to avoid communicating illegal combinations return [trtp.AutoTuneCombination("FP32*FP16, FP32", "LINEAR, LINEAR", [1, 2]), trtp.AutoTuneCombination("FP32, FP32", "HWC, LINEAR", [1, 2])] NcSg|]}|qSr!strip.0sr!r!r" z0AutoTuneCombination.__init__..,LINEARcSr?r!r@rBr!r!r"rErFrr)r r>r'r r_tactics)r)r>r'r(r!r!r"r+ts@ zAutoTuneCombination.__init__rHposr5cCst|t|jkr9|jdgt|dt|j|jdgt|dt|jt|jt|jks9J|D]*}|j|durLtd|d|j|dur[td|d||j|<||j|<q;dS)a Specify I/O types and formats for a specified set of I/O indices. Args: pos (Iterable[int]): I/O indices. Input indices are [0, 1, ..., #inputs - 1] and output indices are [#inputs, #inputs + 1, ..., #inputs + #outputs - 1]. io_types (str): Data types for these I/O indices. layouts (str, optional): Tensor format(s) for these I/O indices. Defaults to "LINEAR". Raises: ValueError: If types or layouts for any of these I/O indices is already specified. .. code-block:: python :linenos: :caption: For a plugin with 3 I/Os, I/O indices 0 and 1 are dependently either FP32/FP16 and index 2 is independently FP32/FP16. @trtp.autotune("my::plugin") def autotune(inp0: trtp.TensorDesc, inp1: trtp.TensorDesc, outputs: Tuple[trtp.TensorDesc]) -> List[trtp.AutoTuneCombination]: c = trtp.AutoTuneCombination() c.pos([0, 1], "FP32|FP16", "LINEAR") c.pos(2, "FP32*FP16") # Omitting format is the same as declaring it to be LINEAR. c.tactics([1, 2]) return [c] NrzType(s) for position z already specifiedzLayout(s) for position )rr r>rr'r )r)rJr>r'pr!r!r"rJs$$  zAutoTuneCombination.poscCs ||_dS)z Specify custom tactics for this type/format combination Args: tactics (Iterable[int]): Custom tactics. These must be positive integers. N)rI)r)r(r!r!r"r(s zAutoTuneCombination.tacticscCstg|_t|j}t|j}|D](}|D]#}tt|j}dd|D|_dd|D|_|j|_|j |qqdS)NcSsg|]}t|qSr!r)rCttr!r!r"rErFz>AutoTuneCombination._generate_combinations..cSsg|]}ttj|qSr!)getattrtrt TensorFormat)rCffr!r!r"rEs) rr#r>r'r$r r&rIr(r)r)type_combinationslayout_combinationstlrr!r!r"_generate_combinationss  z*AutoTuneCombination._generate_combinationscCs||jSr%)rUrr1r!r!r"_get_combinationssz%AutoTuneCombination._get_combinationscCsLtt|jD]}|j|j|t|kr#|j|j||jkr#dSqdS)NTF)rr rr&rr'name)r)rJtypelayoutrr!r!r"_checks zAutoTuneCombination._check)NNN)rH) r:r;r<r8rintr+rJr(rUrVrZr!r!r!r"r=rs S$  r=)r tensorrtrNtypingrrr_utilsr_exportrr#r$r=r!r!r!r"s  7