o ƠiH7@s ddlmZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZddlmZmZddlmZdd lmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:m;Z;mZ>ddl?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGeHeIZJe ddde jKe jLe jMe jNe jOe jPe jQe jRe jSe jTe jUe jVe jWe jXdde jYe jZe j[e j\e j]e j^e j_e j`e jae jbe jce jde jee jfe jge jhe jie jje jke jle jme jne joe jpe jqe jre jse jtd.ddWdXZu ddde jKe jLe jMe jNe jOe jPe jQe jRe jSe jTe jUe jVe jWe jXdde jYe jZe j[e j\e j]e j^e j_e j`e jae jbe jce jde jee jfe jge jhe jie jje jke jle jme jne joe jpe jqe jre jve jwe jxe jye jze j{e j|e jte jsdY5ddfdgZ}e8de-dfddhddrdsZ~ ddde jKe jLe jMe jNe jOe jPe jQe jRe jSe jTe jUe jVe jWe jXdde jYe jZe j[e j\e j]e j^e j_e j`e jae jbe jce jde jee jfe jge jhe jie jje jke jle jme jne joe jpe jqe jrdt,ddvdwZeddzd{Zddd}d~ZdS)) annotationsN)Any CollectionListOptionalSequenceSetTupleUnion)ExportedProgram)Target)Device)EngineCapabilitydtype)needs_cross_compile)Input) _defaults partitioning) DryRunTrackerPerSubgraphDatadryrun_stats_displayparse_non_trt_nodes)BaseEngineCacheDiskEngineCache)!replace_execute_engine_no_op_node)CompilationSettingsUnsupportedOperatorExceptionconvert_moduleinterpret_module_to_resultrepair_double_inputs)DYNAMO_CONVERTERS)DebuggerConfig)fn_supports_debugger)get_decompositions post_loweringpre_export_lowering)resource_partition)deallocate_moduleget_cpu_memory_usageget_flat_args_with_checkget_output_metadataparse_graph_ioprepare_inputsto_torch_deviceto_torch_tensorrt_device). arg_inputs kwarg_inputsdevice disable_tf32assume_dynamic_shape_supportsparse_weightsenabled_precisionsengine_capabilitynum_avg_timing_itersworkspace_size dla_sram_sizedla_local_dram_sizedla_global_dram_sizetruncate_doublerequire_full_compilationmin_block_sizetorch_executed_opstorch_executed_modulespass_through_build_failuresmax_aux_streamsversion_compatibleoptimization_leveluse_python_runtimeuse_fast_partitioner"enable_experimental_decompositionsdryrunhardware_compatibletiming_cache_pathlazy_engine_initcache_built_enginesreuse_cached_enginesengine_cache_direngine_cache_sizecustom_engine_cacheuse_explicit_typing use_fp32_accrefit_identical_engine_weightsstrip_engine_weightsimmutable_weightsenable_weight_streamingtiling_optimization_levell2_limit_for_tilingoffload_module_to_cpuuse_distributed_mode_traceenable_resource_partitioningcpu_memory_budgetexported_programr inputs!Optional[Sequence[Sequence[Any]]]r/r0Optional[dict[Any, Any]]r1*Optional[Union[Device, torch.device, str]]r2boolr3r4r5GUnion[Set[Union[torch.dtype, dtype]], Tuple[Union[torch.dtype, dtype]]]r6rr7intr8r9r:r;r<r=r>r?Optional[Collection[Target]]r@Optional[List[str]]rArB Optional[int]rCrDrErFrGrHrIrJstrrKrLrMrNrOrPOptional[BaseEngineCache]rQrRrSrTrUrVrWrXrYrZr[r\kwargsrreturntorch.fx.GraphModulec.< stdkstddkrtdtdtd|0ddr,tjdtd d d |0vrG|t j ur;t d |0d }tjd td d d|0vr`tjdtd d |(r[t d|0d }(d|0vrytjdtd d |(rtt d|0d }(|&r|(rt dt | } |dur|rtd|d|$rtdkstfddtjtjtjtjhDstdd|%rtd|)r|$std|dur|dur|durtd|dur|durtd |p|}|duri}t|tjjs|g}t|}1t|}2t|}d!d"Did#rnt j d$|d%|d&| d'|d(|dur(|nt!d)|d*|d+|d,|d-dd.|d/|d0| d1|d2|d3|id4|d5| d6| d7| d8|d9|d:|d;|d<|d=|d>| d?|&d@|'dA|(dBdCdD|)dE|*|+|-|.|/dF}3dG}4|3"D]\}5}6|5|4vr|6rd|3|5<tdH|5dIqt#dPi|3}7t$dJ|7t%||7}|&t'|}|(}8tdKt)|8j*t+|8|7}8tdLt)|8j*|,rt,|(ddMt$dNntj-.\}9}:|9|:d krtdOt/|8|1|2|7};|;S)Qa&#Compile an ExportedProgram module using TensorRT in Linux for Inference in Windows Takes an exported program and a set of settings to configure the compiler and it will convert methods to AOT graphs which call equivalent TensorRT engines Arguments: exported_program (torch.export.ExportedProgram): Source module, running torch.export on a ``torch.nn.Module`` inputs (Tuple[Any, ...]): List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. .. code-block:: py inputs=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] Keyword Arguments: arg_inputs (Tuple[Any, ...]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (dict[Any, ...]): Optional, kwarg inputs to the module forward function. device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on :: device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True) disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas assume_dynamic_shape_support (bool): Setting this to true enables the converters work for both dynamic and static shapes. Default: False sparse_weights (bool): Enable sparsity for convolution and fully connected layers. enabled_precisions (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels workspace_size (int): Maximum size of workspace given to TensorRT dla_sram_size (int): Fast software managed RAM used by DLA to communicate within a layer. dla_local_dram_size (int): Host RAM used by DLA to share intermediate tensor data across operations dla_global_dram_size (int): Host RAM used by DLA to store weights and metadata for execution truncate_double (bool): Truncate weights provided in double (float64) to float32 require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch min_block_size (int): The minimum number of contiguous TensorRT convertible operations in order to run a set of operations in TensorRT torch_executed_ops (Collection[Target]): Set of aten operators that must be run in PyTorch. An error will be thrown if this set is not empty but ``require_full_compilation`` is True torch_executed_modules (List[str]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True pass_through_build_failures (bool): Error out if there are issues during compilation (only applicable to torch.compile workflows) max_aux_stream (Optional[int]): Maximum streams in the engine version_compatible (bool): Build the TensorRT engines compatible with future versions of TensorRT (Restrict to lean runtime operators to provide version forward compatibility for the engines) optimization_level: (Optional[int]): Setting a higher optimization level allows TensorRT to spend longer engine building time searching for more optimization options. The resulting engine may have better performance compared to an engine built with a lower optimization level. The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is currently 5. Setting it to be greater than the maximum level results in identical behavior to the maximum level. use_python_runtime: (bool): Return a graph using a pure Python runtime, reduces options for serialization use_fast_partitioner: (bool): Use the adjacency based partitioning scheme instead of the global partitioner. Adjacency partitioning is faster but may not be optimal. Use the global paritioner (``False``) if looking for best performance enable_experimental_decompositions (bool): Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT. dryrun (bool): Toggle for "Dryrun" mode, running everything except conversion to TRT and logging outputs hardware_compatible (bool): Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer) timing_cache_path (str): Path to the timing cache if it exists (or) where it will be saved after compilation lazy_engine_init (bool): Defer setting up engines until the compilation of all engines is complete. Can allow larger models with multiple graph breaks to compile but can lead to oversubscription of GPU memory at runtime. cache_built_engines (bool): Whether to save the compiled TRT engines to storage reuse_cached_engines (bool): Whether to load the compiled TRT engines from storage engine_cache_dir (Optional[str]): Directory to store the cached TRT engines engine_cache_size (Optional[int]): Maximum hard-disk space (bytes) to use for the engine cache, default is 1GB. If the cache exceeds this size, the oldest engines will be removed by default custom_engine_cache (Optional[BaseEngineCache]): Engine cache instance to use for saving and loading engines. Users can provide their own engine cache by inheriting from BaseEngineCache. If used, engine_cache_dir and engine_cache_size will be ignored. use_explicit_typing (bool): This flag enables strong typing in TensorRT compilation which respects the precisions set in the Pytorch model. This is useful when users have mixed precision graphs. use_fp32_acc (bool): This option inserts cast to FP32 nodes around matmul layers and TensorRT ensures the accumulation of matmul happens in FP32. Use this only when FP16 precision is configured in enabled_precisions. refit_identical_engine_weights (bool): Refit engines with identical weights. This is useful when the same model is compiled multiple times with different inputs and the weights are the same. This will save time by reusing the same engine for different inputs. strip_engine_weights (bool): Strip engine weights from the serialized engine. This is useful when the engine is to be deployed in an environment where the weights are not required. immutable_weights (bool): Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, `strip_engine_weights` and `refit_identical_engine_weights` will be ignored. enable_weight_streaming (bool): Enable weight streaming. tiling_optimization_level (str): The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support ["none", "fast", "moderate", "full"]. l2_limit_for_tiling (int): The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit). use_distributed_mode_trace (bool): Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model enable_resource_partitioning (bool): Enable resource-aware partitioning. This is useful when the model is large and the CPU memory is limited. cpu_memory_budget (Optional[int]): The maximum amount of CPU memory to use for the compilation. If the compilation requires more memory than this budget, the compilation will fail. **kwargs: Any, Returns: torch.fx.GraphModule: Compiled FX Module, when run it will execute via TensorRT Linuxr64bitznCross compile for windows is only supported on x86-64 Linux architecture, current platform: platform.system()=z, platform.architecture()[0]=debugF`debug` is deprecated. Please use `with torch_tensorrt.dynamo.Debugger(...)` to wrap your compilation call to enable debugging functionality. stackleveltruncate_long_and_double}Provided configuration for "truncate_double" and deprecated API "truncate_long_and_double", please only use "truncate_double"Compiler option "truncate_long_and_double" is deprecated in favor of "truncate_double" as int64 is now natively supported, this option will be removed in the next versionrefitzw`refit` is deprecated. Please set `immutable_weights=False` to build a refittable engine whose weights can be refitted.>Use flag `immutable_weights` only. Flag `refit` is deprecated.make_refittable`make_refittable` is deprecated. Please set `immutable_weights=False` to build a refittable engine whose weights can be refittedHUse flag `immutable_weights` only. Flag `make_refittable` is deprecated.P`immutable_weights` must be False when `refit_identical_engine_weights` is True.N/Detected torch_executed_modules was non-empty: = This feature is unimplemented in Torch-TRT Dynamo currently.c3|]}|vVqdSN.0xr5rS/home/ubuntu/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/_compiler.py   z,cross_compile_for_windows..duse_explicit_typing was set to True, however found that enabled_precisions was also specified (saw: e, expected: dtype.f32, dtype.f4). enabled_precisions should not be used when use_explicit_typing=True4FP32 accumulation for matmul layers is enabled. This option should only be enabled if the model already has FP16 weights and has no effect if it has FP32 weights. This flag inserts casts around matmul layers and ensures TensorRT executes the matmul layers in FP16 with FP32 accumulation.ZWhen enable_weight_streaming is enabled, it requires use_explicit_typing to be set to TrueA'arg_inputs', 'kwarg_inputs' and 'inputs' should not all be None.>'arg_inputs' and 'inputs' should not be used at the same time.cSh|]}t|qSrr_fromrprrr -z,cross_compile_for_windows..r5r1r3r8r>r?rArBrCrDrEr<rFr7rGr=r2r4r6r9r:r;rHrIrJrKrLrMrSrTrU enable_cross_compile_for_windowsTrVrW)rXrZr[r\)rErKrLrMzarg: zR is not supported for cross compilation for windows feature, hence it is disabled.Compilation Settings: %s Input graph: Lowered Input graph:  delete_moduleThe PyTorch model was moved to the CPU to allocate all GPU memory to TensorRT. To retain the model on the GPU, set offload_module_to_cpu=FalseRemaining GPU memory may not be enough to compile the TensorRT engine for this model resulting in an OOM error, Consider setting offload_module_to_cpu=Truer)0platformsystem architecture RuntimeErrorgetwarningswarnDeprecationWarningkeysrTRUNCATE_DOUBLE ValueErrorrrloggerwarninglenanytorchfloat32rf32float4_e2m1fn_x2f4AssertionErrorro isinstance collectionsabcrr,r.ENABLED_PRECISIONSsetitemsrinfor%run_decompositionsr#modulerhgraphr$r'cuda mem_get_infocompile_module)r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\rjtrt_arg_inputstrt_kwarg_inputscompilation_optionsunsupported_settingskeyvaluesettingsgmremaining_memory total_memorytrt_gmrrrcross_compile_for_windows<s               !"#$%&'.    r)5r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZenable_autocastautocast_low_precision_typeautocast_excluded_nodesautocast_excluded_opsautocast_max_output_thresholdautocast_max_depth_of_reductionautocast_calibration_dataloaderr\r[rr#Optional[Union[torch.dtype, dtype]]rCollection[str]rCollection[Target]rfloatrr%Optional[torch.utils.data.DataLoader]c5A s|7ddrtjdtdd|7ddstjdtddd|7vr7|tjur+td |7d}tjd tddd |7vrPtjd tdd|(rKtd |7d  }(d|7vritjdtdd|(rdtd|7d }(|&rq|(rqtdd|7vr|7drtdt | } |dur|rt d|d|.rd}$t d|$rt dkstfddtjtjtjtjhDstdd|/durt|/tjtfstdt|/|/tjtjhvr|/tjtjhvrtd|/|%rt d |)r|$std!|dur |dur |dur td"|dur|durtd#|p|}|dur'i}t|tjjs2|g}t|}8t|}9t |}d$d%Dd}:|sM| rY|#durT|#nt!|!|"}:id&r`ntj"d'|d(|d)| d*|d+|durw|nt#d,|d-|d.|d/|d0|d1|d2|d3| d4|d5|d6|id7|d8| d9| d:| d;|d<|d=|d>|d?|d@|dA| d|$dB|%dC|&dD|'dE|(dd|)|*|+|,|-|.|/|0|1|2|3|4|6|5dF};t dGt$dHt%dQi|;}}?|>|?dkr`t dPt1|=|8|9|<|:}@|@S)RaR)Compile an ExportedProgram module for NVIDIA GPUs using TensorRT Takes a existing TorchScript module and a set of settings to configure the compiler and will convert methods to JIT Graphs which call equivalent TensorRT engines Converts specifically the forward method of a TorchScript Module Arguments: exported_program (torch.export.ExportedProgram): Source module, running torch.export on a ``torch.nn.Module`` inputs (Optional[Sequence[Sequence[Any]]]): List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. .. code-block:: py inputs=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] Keyword Arguments: arg_inputs (Optional[Sequence[Sequence[Any]]]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (Optional[dict[Any, Any]]): kwarg inputs to the module forward function. device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on :: device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True) disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas assume_dynamic_shape_support (bool): Setting this to true enables the converters work for both dynamic and static shapes. Default: False sparse_weights (bool): Enable sparsity for convolution and fully connected layers. enabled_precisions (Union[Set[Union[torch.dtype, dtype]], Tuple[Union[torch.dtype, dtype]]]): The set of datatypes that TensorRT can use when selecting kernels engine_capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels workspace_size (int): Maximum size of workspace given to TensorRT dla_sram_size (int): Fast software managed RAM used by DLA to communicate within a layer. dla_local_dram_size (int): Host RAM used by DLA to share intermediate tensor data across operations dla_global_dram_size (int): Host RAM used by DLA to store weights and metadata for execution truncate_double (bool): Truncate weights provided in double (float64) to float32 require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch min_block_size (int): The minimum number of contiguous TensorRT convertible operations in order to run a set of operations in TensorRT torch_executed_ops (Optional[Collection[Target]]): Set of aten operators that must be run in PyTorch. An error will be thrown if this set is not empty but ``require_full_compilation`` is True torch_executed_modules (Optional[List[str]]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True pass_through_build_failures (bool): Error out if there are issues during compilation (only applicable to torch.compile workflows) max_aux_streams (Optional[int]): Maximum streams in the engine version_compatible (bool): Build the TensorRT engines compatible with future versions of TensorRT (Restrict to lean runtime operators to provide version forward compatibility for the engines) optimization_level: (Optional[int]): Setting a higher optimization level allows TensorRT to spend longer engine building time searching for more optimization options. The resulting engine may have better performance compared to an engine built with a lower optimization level. The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is currently 5. Setting it to be greater than the maximum level results in identical behavior to the maximum level. use_python_runtime: (bool): Return a graph using a pure Python runtime, reduces options for serialization use_fast_partitioner: (bool): Use the adjacency based partitioning scheme instead of the global partitioner. Adjacency partitioning is faster but may not be optimal. Use the global paritioner (``False``) if looking for best performance enable_experimental_decompositions (bool): Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT. dryrun (bool): Toggle for "Dryrun" mode, running everything except conversion to TRT and logging outputs hardware_compatible (bool): Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer) timing_cache_path (str): Path to the timing cache if it exists (or) where it will be saved after compilation lazy_engine_init (bool): Defer setting up engines until the compilation of all engines is complete. Can allow larger models with multiple graph breaks to compile but can lead to oversubscription of GPU memory at runtime. cache_built_engines (bool): Whether to save the compiled TRT engines to storage reuse_cached_engines (bool): Whether to load the compiled TRT engines from storage engine_cache_dir (str): Directory to store the cached TRT engines engine_cache_size (int): Maximum hard-disk space (bytes) to use for the engine cache, default is 1GB. If the cache exceeds this size, the oldest engines will be removed by default custom_engine_cache (Optional[BaseEngineCache]): Engine cache instance to use for saving and loading engines. Users can provide their own engine cache by inheriting from BaseEngineCache. If used, engine_cache_dir and engine_cache_size will be ignored. use_explicit_typing (bool): This flag enables strong typing in TensorRT compilation which respects the precisions set in the Pytorch model. This is useful when users have mixed precision graphs. use_fp32_acc (bool): This option inserts cast to FP32 nodes around matmul layers and TensorRT ensures the accumulation of matmul happens in FP32. Use this only when FP16 precision is configured in enabled_precisions. refit_identical_engine_weights (bool): Refit engines with identical weights. This is useful when the same model is compiled multiple times with different inputs and the weights are the same. This will save time by reusing the same engine for different inputs. strip_engine_weights (bool): Strip engine weights from the serialized engine. This is useful when the engine is to be deployed in an environment where the weights are not required. immutable_weights (bool): Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, `strip_engine_weights` and `refit_identical_engine_weights` will be ignored. enable_weight_streaming (bool): Enable weight streaming. tiling_optimization_level (str): The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support ["none", "fast", "moderate", "full"]. l2_limit_for_tiling (int): The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit). offload_module_to_cpu (bool): Offload the module to CPU. This is useful when we need to minimize GPU memory usage. use_distributed_mode_trace (bool): Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model enable_autocast (bool): Whether to enable autocast. If enabled, use_explicit_typing will be set to True. autocast_low_precision_type (Optional[Union[torch.dtype, dtype]]): The precision to reduce to. We currently support torch.float16 and torch.bfloat16. Default is None, which means no low precision is used. autocast_excluded_nodes (Collection[str]): The set of regex patterns to match user-specified node names that should remain in FP32. Default is []. autocast_excluded_ops (Collection[Target]): The set of targets (ATen ops) that should remain in FP32. Default is []. autocast_max_output_threshold (float): Maximum absolute value for node outputs, nodes with outputs greater than this value will remain in FP32. Default is 512. autocast_max_depth_of_reduction (Optional[int]): Maximum depth of reduction allowed in low precision. Nodes with higher reduction depths will remain in FP32. This helps prevent excessive accuracy loss in operations particularly sensitive to reduced precision, as higher-depth reductions may amplify computation errors in low precision formats. If not provided, infinity will be used. Default is None. autocast_calibration_dataloader (Optional[torch.utils.data.DataLoader]): The dataloader to use for autocast calibration. Default is None. enable_resource_partitioning (bool): Enable resource-aware partitioning. This is useful when the model is large and the CPU memory is limited. cpu_memory_budget (Optional[int]): The maximum amount of CPU memory to use for the compilation. If the compilation requires more memory than this budget, the compilation will fail. **kwargs: Any, Returns: torch.fx.GraphModule: Compiled FX Module, when run it will execute via TensorRT roFrprqrrrQzi`use_explicit_typing` is deprecated. This setting will be removed and you should enable autocast instead.rtrurvrwv`refit` is deprecated. Please set `immutable_weights=False` to build a refittable engine whose weights can be refittedrxryrzr{r|rPlease use torch_tensorrt.dynamo.cross_compile_for_windows() if you want to cross compile the module in Linux for inferencing in Windows.Nr}r~Tz9Autocast is enabled, setting use_explicit_typing to True.rc3rrrrrrrryrzcompile..rrzVautocast_low_precision_type must be a torch.dtype or torch_tensorrt._enums.dtype, got zeautocast_low_precision_type must be one of torch.float16, torch.bfloat16, dtype.f16, dtype.bf16, got rrrrcSrrrrrrrrrzcompile..r5r1r3r8r>r?rArBrCrDrEr<rFr7rGr=r2r4r6r9r:r;rHrIrJrKrLrMrRrSrTrU)rVrWrXrYrZrrrrrrrr[r\z"CPU memory usage before lowering: z MBrrz&CPU memory usage after post_lowering: rrrz$CPU memory usage after CPU offload: rr)2rrrrrrrrrrrrrorrrrrrrrrrtypefloat16bfloat16f16bf16rrrr,r.rrrr(rrr%rr#rrhrr$r'rrr)Ar]r^r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZrrrrrrrr\r[rjrr engine_cacherrrrrrrrrcompiles                       !"#$%&'6     r)_debugger_configrsample_arg_inputsSequence[Input]sample_kwarg_inputsrrrrOptional[DebuggerConfig]c# stdd|Drtdt}|duri}t|t ||j \}}||_ ||_ ||_ |jr?|jdkr?td|dksK||jkr[|js[t|d|d |j|Std |d |d d>dd} | |sutdt|dd} t|dd} d?dd} d} |jrztdtj||j|j |j||kd\}}Wntjjjjytjd d!d"d!} d|_Ynw|jstd#tj||j|j |jd$\}}|jrt||jd%}|j |_!|js|j"#t$|i}|j%j&D] }d&|j'vrq|||j'<q| | | |i}t(|D]}|)d'rt*||q |+D]L\}}t||}t,|tjj-j.s4q|jr[d&|vr[|j"#t$|td(t/|t/|j%|0t1|j2q||vrht3d)|d*t4|}t5|dksuJd+d,g}|D]&||j6vrfd-d.|D}|||j6<td/|d0nq{t7}||_8t5d1d.|j%j&D|_9t:|}|dusJtd2t/|d3d.|Dt/|j%|j;rt<|||t1|j2|}t=|||j>d7_>|j?@|tjAB|jsktC|||||d4}|||<|rk|jDrD|jEr#|jFd5krt3d6|Gn!|jFd5kr-t3d7tHjIJ|jKd8}tHjL|d!d9|jG||jFd:|jMrktNtHjIJ|jKd;d<}|O|PWdn 1sfwYqt=|||QD]\}}tR||||jSr|jTst||}|UqvtV|j%j&d=}|jWD]} | D]}!|!jX}"d&t/|"vrqt||"Yd!qq| rd!|_tZ||j|S)@awCompile a traced FX module Includes: Partitioning + Conversion Phases Args: module: FX GraphModule to convert arg_inputs: Inputs to the module kwarg_inputs: kwargs to the module settings: Compilation settings engine_cache: Engine cache instance to store/load compiled engines Returns: Compiled FX GraphModule css|]}|jVqdSr) requires_grad)rvrrrr+sz!compile_module..zZThe model may be in training mode, which may affect the performance of the compiled model!Nrz^It is recommended to run `dryrun` mode with `min_block_size=1`, for the most thorough analysisrz6 supported operations detected in subgraph containing zV computational nodes. Skipping this subgraph, since min_block_size was detected to be zDetected support for z operators out of z in subgraph.rrlrkrbcSsN|jjD] }|jdkr$|js$d|jvr$td|jd|jddSqdS)NoutputvalzNode z of op type zI does not have metadata. This could sometimes lead to undefined behavior.FT)rnodesopmetarrname)rnoderrrcontains_metadataTs z)compile_module..contains_metadatazSome nodes do not have metadata (shape and dtype information). This could lead to problems sometimes if the graph has PyTorch and TensorRT segments._in_spec _out_specin_specrout_spec target_moduleNonecSs$|dur||_|dur||_dSdS)z Applies input and output specs to the target module. Args: in_spec: The input spec to apply out_spec: The output spec to apply target_module: The module to apply specs to N)rr)rrrrrrpreserve_module_specsis   z-compile_module..preserve_module_specsFz/Partitioning the graph via the fast partitioner)r>r?r= skip_fusionziPartitioning failed on the subgraph with fast partition. See trace above. Retrying with global partition.Texc_infoz1Partitioning the graph via the global partitioner)r>r?r=)r\ _run_on_acc _frozen_paramzSubmodule in PyTorch: %s %sz node_name: z0 does not exist in the submodule node dictionaryr tensor_metacsg|] }|vr|qSrr)rmetadatarrr sz"compile_module..zUpdated metadata for node: z) with its corresponding submodule outputscSsg|] }|jdvr|qS)) call_function call_method call_module)r)rrrrrrs  z.Converting submodule: %s Input shapes: %s %scSsg|]}|jqSr)shape)rinputrrrrs)rrr cudagraphzProfiling with TREX can only be enabled when using the C++ runtime. Python runtime profiling only support cudagraph visualization.zProfiling with Cudagraph can only be enabled when using the Python runtime. C++ runtime profiling only support TREX/Perfetto visualization.engine_visualization_profile)exist_ok)profiling_results_dirprofile_formatzengine_layer_info.jsonw)rrlrkrb)rrrrrrlrkr)[r state_dictvaluesrrr CONVERTERSset_compilation_settingsrget_graph_converter_supportr?total_ops_in_graphsupported_ops_in_graphcompilation_settingsrHr>rrogetattrrFfast_partitionr=rfxpasses splitter_baseFxNetSplitterInternalErrorerrorglobal_partitionr[r&r\unsupported_operatorsunsupported_opsto_run_in_torchextendrrrrdir startswithdelattrnamed_childrenr graph_module GraphModulerhtor-r1rr*rrr subgraph_namesubgraph_op_countconstruct_submodule_inputsr<rr+tensorrt_graph_countper_subgraph_dataappendr empty_cachersave_engine_profilerEr enable_profilingospathjoin logging_dirmakedirssave_layer_infoopenwriteget_layer_inforsetattrrKr setup_enginelistargstargetset_output_tensors_as_unownedr)#rrrrrrdryrun_trackernum_supported_ops total_opsroriginal_in_specoriginal_out_specrfast_partitioner_failedpartitioned_module supported_opssubmodule_node_dictr trt_modulesattrr_ submodule metadata_list metadata_keys meta_val_list subgraph_datasubmodule_inputs trt_moduler2f output_nodeargrr>rrrrs                                     r),r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZbytesc,; sf|.ddrtjdtddd|.vr)|tjurtd|.d}tjdtddd |.vrBtjd tdd|(r=td |.d  }(d |.vr[tjd tdd|(rVtd|.d  }(|&rc|(rctdd|.vrq|.drqtdt | } |dur|rt d|d|$rt dkst fddtjtjtjtjhDstdd|%rt d|)r|$std|dur|dur|durtd|dur|durtd|p|}|duri}t|tjjs|g}t|}/t|}0t|}ddDd}1|s| r|#dur |#nt|!|"}1id rntjd!|d"|d#| d$|d%|dur,|ntd&|d'|d(|d)|d*|d+|d,|d-| d.|d/|d0|id1|d2| d3| d4| d5|d6|d7|d8|d9|d:|d;| d<|$d=|%d>|&d?|'d@|(dd|)|*|+|,|-dA}2tdOi|2}3t dB|3t!||3}|"t#|}|$}4t dCt%|4j&t'|4|3}4t dDt%|4j&|,rt(|$ddEt dFntj)*\}5}6|5|6dkrt dGt+|t,|/|0dH}7z t-|4|7|3|1dI}8Wn1t.yt j/dJ|4dKdLdMYnt0y-}9zt j/dN|9dLdMWYd}9~9nd}9~9ww|8j1}:|:S)PaU"Convert an ExportedProgram to a serialized TensorRT engine Converts an ExportedProgram to a serialized TensorRT engine given a dictionary of conversion settings Arguments: exported_program (torch.export.ExportedProgram): Source module, running torch.export on a ``torch.nn.Module`` inputs (Optional[Sequence[Sequence[Any]]]): List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. .. code-block:: py inputs=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] Keyword Arguments: arg_inputs (Optional[Sequence[Sequence[Any]]]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (Optional[dict[Any, Any]]): kwarg inputs to the module forward function. device (Union(torch_tensorrt.Device, torch.device, dict)): Target device for TensorRT engines to run on :: device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True) disable_tf32 (bool): Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas assume_dynamic_shape_support (bool): Setting this to true enables the converters work for both dynamic and static shapes. Default: False sparse_weights (bool): Enable sparsity for convolution and fully connected layers. enabled_precisions (Union[Set[Union[torch.dtype, dtype]], Tuple[Union[torch.dtype, dtype]]]): The set of datatypes that TensorRT can use when selecting kernels engine_capability (torch_tensorrt.EngineCapability): Restrict kernel selection to safe gpu kernels or safe dla kernels num_avg_timing_iters (int): Number of averaging timing iterations used to select kernels workspace_size (int): Maximum size of workspace given to TensorRT dla_sram_size (int): Fast software managed RAM used by DLA to communicate within a layer. dla_local_dram_size (int): Host RAM used by DLA to share intermediate tensor data across operations dla_global_dram_size (int): Host RAM used by DLA to store weights and metadata for execution truncate_double (bool): Truncate weights provided in double (float64) to float32 require_full_compilation (bool): Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch min_block_size (int): The minimum number of contiguous TensorRT convertible operations in order to run a set of operations in TensorRT torch_executed_ops (Optional[Collection[Target]]): Set of aten operators that must be run in PyTorch. An error will be thrown if this set is not empty but ``require_full_compilation`` is True torch_executed_modules (Optional[List[str]]): List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but ``require_full_compilation`` is True pass_through_build_failures (bool): Error out if there are issues during compilation (only applicable to torch.compile workflows) max_aux_streams (Optional[int]): Maximum streams in the engine version_compatible (bool): Build the TensorRT engines compatible with future versions of TensorRT (Restrict to lean runtime operators to provide version forward compatibility for the engines) optimization_level: (Optional[int]): Setting a higher optimization level allows TensorRT to spend longer engine building time searching for more optimization options. The resulting engine may have better performance compared to an engine built with a lower optimization level. The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is currently 5. Setting it to be greater than the maximum level results in identical behavior to the maximum level. use_python_runtime: (bool): Return a graph using a pure Python runtime, reduces options for serialization use_fast_partitioner: (bool): Use the adjacency based partitioning scheme instead of the global partitioner. Adjacency partitioning is faster but may not be optimal. Use the global paritioner (``False``) if looking for best performance enable_experimental_decompositions (bool): Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT. dryrun (bool): Toggle for "Dryrun" mode, running everything except conversion to TRT and logging outputs hardware_compatible (bool): Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer) timing_cache_path (str): Path to the timing cache if it exists (or) where it will be saved after compilation lazy_engine_init (bool): Defer setting up engines until the compilation of all engines is complete. Can allow larger models with multiple graph breaks to compile but can lead to oversubscription of GPU memory at runtime. cache_built_engines (bool): Whether to save the compiled TRT engines to storage reuse_cached_engines (bool): Whether to load the compiled TRT engines from storage engine_cache_dir (str): Directory to store the cached TRT engines engine_cache_size (int): Maximum hard-disk space (bytes) to use for the engine cache, default is 1GB. If the cache exceeds this size, the oldest engines will be removed by default custom_engine_cache (Optional[BaseEngineCache]): Engine cache instance to use for saving and loading engines. Users can provide their own engine cache by inheriting from BaseEngineCache. If used, engine_cache_dir and engine_cache_size will be ignored. use_explicit_typing (bool): This flag enables strong typing in TensorRT compilation which respects the precisions set in the Pytorch model. This is useful when users have mixed precision graphs. use_fp32_acc (bool): This option inserts cast to FP32 nodes around matmul layers and TensorRT ensures the accumulation of matmul happens in FP32. Use this only when FP16 precision is configured in enabled_precisions. refit_identical_engine_weights (bool): Refit engines with identical weights. This is useful when the same model is compiled multiple times with different inputs and the weights are the same. This will save time by reusing the same engine for different inputs. strip_engine_weights (bool): Strip engine weights from the serialized engine. This is useful when the engine is to be deployed in an environment where the weights are not required. immutable_weights (bool): Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, `strip_engine_weights` and `refit_identical_engine_weights` will be ignored. enable_weight_streaming (bool): Enable weight streaming. tiling_optimization_level (str): The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support ["none", "fast", "moderate", "full"]. l2_limit_for_tiling (int): The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit). offload_module_to_cpu (bool): Offload the module to CPU. This is useful when we need to minimize GPU memory usage. use_distributed_mode_trace (bool): Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model. **kwargs: Any, Returns: bytes: Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs roFrprqrrrtrurvrwrrxryrzr{r|rrNr}r~rc3rrrrrrrrrzDconvert_exported_program_to_serialized_trt_engine..rrrrrrcSrrrrrrrr=rzDconvert_exported_program_to_serialized_trt_engine..r5r1r3r8r>r?rArBrCrDrEr<rFr7rGr=r2r4r6r9r:r;rHrIrJrKrLrMrQrRrSrTrU)rVrWrXrYrZrrrrrrr)r^rrzConversion of module z. not currently fully supported or convertible!Trz,While interpreting the module got an error: r)2rrrrrrrrrrrrrrrrrrrrrrorrrrr,r.rrrrrr%rr#rrhrr$r'rrr)r<rrr Exceptionserialized_engine);r]r^r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZrjrrrrrrrrflattened_input_listinterpreter_resulterXrrr1convert_exported_program_to_serialized_trt_engineNs                  !"#$%&'.       r\ file_pathrcCsF|stdddlm}||dd}tj||td|dS)z Save cross compiled exported program to disk. Arguments: module (torch.fx.GraphModule): Cross compiled Torch-TensorRT module file_path (str): the file path where the exported program will be saved to disk ;File path cannot be empty. Please provide a valid file pathr)exportT)cross_compile_modulez-successfully saved the module for windows at N)rtorch_tensorrt.dynamo._exporterr_rsaverro)rr]r_ exp_programrrr$save_cross_compiled_exported_programs    rdc Cs|stdtdkstdkrtdztd|dtj|}Wt |St yK}ztj d|d|dd td |d d }~ww) a% Load an ExportedProgram file in Windows which was previously cross compiled in Linux Arguments: file_path (str): Path to file on the disk Raises: ValueError: If the api is not called in windows or there is no file or the file is a valid ExportedProgram file r^WindowsAMD64zMcross runtime compiled model for windows can only be loaded in Windows systemzLoading the provided file z using torch.export.load()z: via torch.export.load() failed with the following error: Trzcross_load the file zL doesn't correspond to a valid ExportedProgram. Please verify the file path.N) rrrmachinerrorr_loadrWrr)r]rcr[rrr$load_cross_compiled_exported_programs(   rjr)dr]r r^r_r/r_r0r`r1rar2rbr3rbr4rbr5rcr6rr7rdr8rdr9rdr:rdr;rdr<rbr=rbr>rdr?rer@rfrArbrBrgrCrbrDrgrErbrFrbrGrbrHrbrIrbrJrhrKrbrLrbrMrbrNrhrOrdrPrirQrbrRrbrSrbrTrbrUrbrVrbrWrhrXrdrYrbrZrbr[rbr\rgrjrrkrl)rr]r r^r_r/r_r0r`r1rar2rbr3rbr4rbr5rcr6rr7rdr8rdr9rdr:rdr;rdr<rbr=rbr>rdr?rer@rfrArbrBrgrCrbrDrgrErbrFrbrGrbrHrbrIrbrJrhrKrbrLrbrMrbrNrhrOrdrPrirQrbrRrbrSrbrTrbrUrbrVrbrWrhrXrdrYrbrZrbrrbrrrrrrrrrrgrrr\rgr[rbrjrrkrl)rrlrrrr`rrrrirrrkrl)`r]r r^r_r/r_r0r`r1rar2rbr3rbr4rbr5rcr6rr7rdr8rdr9rdr:rdr;rdr<rbr=rbr>rdr?rer@rfrArbrBrgrCrbrDrgrErbrFrbrGrbrHrbrIrbrJrhrKrbrLrbrMrbrNrhrOrdrPrirQrbrRrbrSrbrTrbrUrbrVrbrWrhrXrdrYrbrZrbrjrrkrV)rrlr]rhrkr)re)r]rhrkr) __future__rcollections.abcrloggingr1rrtypingrrrrrrr r r torch.exportr torch.fx.noder torch_tensorrt._Devicer torch_tensorrt._enumsrrtorch_tensorrt._featuresrtorch_tensorrt._Inputrtorch_tensorrt.dynamorr$torch_tensorrt.dynamo._DryRunTrackerrrrr#torch_tensorrt.dynamo._engine_cacherrrar torch_tensorrt.dynamo.conversionrrrrr3torch_tensorrt.dynamo.conversion._ConverterRegistryr r+torch_tensorrt.dynamo.debug._DebuggerConfigr!.torch_tensorrt.dynamo.debug._supports_debuggerr"torch_tensorrt.dynamo.loweringr#r$r%8torch_tensorrt.dynamo.partitioning._resource_partitionerr&torch_tensorrt.dynamo.utilsr'r(r)r*r+r,r-r. getLogger__name__rDEVICE DISABLE_TF32ASSUME_DYNAMIC_SHAPE_SUPPORTSPARSE_WEIGHTSrENGINE_CAPABILITYNUM_AVG_TIMING_ITERSWORKSPACE_SIZE DLA_SRAM_SIZEDLA_LOCAL_DRAM_SIZEDLA_GLOBAL_DRAM_SIZErREQUIRE_FULL_COMPILATIONMIN_BLOCK_SIZEPASS_THROUGH_BUILD_FAILURESMAX_AUX_STREAMSVERSION_COMPATIBLEOPTIMIZATION_LEVELUSE_PYTHON_RUNTIMEUSE_FAST_PARTITIONER"ENABLE_EXPERIMENTAL_DECOMPOSITIONSDRYRUNHARDWARE_COMPATIBLETIMING_CACHE_PATHLAZY_ENGINE_INITCACHE_BUILT_ENGINESREUSE_CACHED_ENGINESENGINE_CACHE_DIRENGINE_CACHE_SIZECUSTOM_ENGINE_CACHEUSE_EXPLICIT_TYPING USE_FP32_ACCREFIT_IDENTICAL_ENGINE_WEIGHTSSTRIP_ENGINE_WEIGHTSIMMUTABLE_WEIGHTSENABLE_WEIGHT_STREAMINGTILING_OPTIMIZATION_LEVELL2_LIMIT_FOR_TILINGOFFLOAD_MODULE_TO_CPUUSE_DISTRIBUTED_MODE_TRACEENABLE_RESOURCE_PARTITIONINGCPU_MEMORY_BUDGETrENABLE_AUTOCASTAUTOCAST_LOW_PRECISION_TYPEAUTOCAST_EXCLUDED_NODESAUTOCAST_EXCLUDED_OPSAUTOCAST_MAX_OUTPUT_THRESHOLDAUTOCAST_MAX_DEPTH_OF_REDUCTIONAUTOCAST_CALIBRATION_DATALOADERrrr\rdrjrrrrs (          ( T =_