o Ơi@sddlZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZddlmZejGd d d ZejGd d d eZdS) N)ListOptionalSetType)nn) PassManager)InputTensorSpec)fuse_permute_linearfuse_permute_matmul)LowerPrecisionc@seZdZUdZdZeed<ejZ eed<dZ eed<dZ e e eejed<dZe e eejed <d Zeed <d Zeed <dS) LowerSettingBasica Basic class for lowering. max_batch_size: The maximum batch size for lowering job. If run with TensorRT lowering, this is the maximum batch size which can be used at execution time, and also the batch size for which the ICudaEngine will be optimized. If run with AITemplate lowering, this the max batch_size for the model. lower_precision: lower precision dtype during lowering. min_acc_module_size(int): minimal number of nodes for an accelerate submodule. ast_rewriter_allow_list (Optional[Set[nn.Module]]): Optional allow list of modules that need AST rewriting. This is aiming to eliminate input variable involve in exception checking control flow. leaf_module_list (Optional[Set[nn.Module]]): Optional leaf module list where modules will not be traced into. verbose_profile (bool): verbosity of profiler, default to False. imax_batch_sizelower_precision min_acc_module_sizeNast_rewriter_allow_listleaf_module_listFverbose_profileis_aten)__name__ __module__ __qualname____doc__rint__annotations__r FP32rrrrrrrModulerrboolrrrS/home/ubuntu/.local/lib/python3.10/site-packages/torch_tensorrt/fx/lower_setting.pyr s    r c@seZdZUdZejedZee e d<dZ e e d<dZ e e d<dZee d <dZe e d <ejd d dZee d <ejdd dZee d<dZe e d<dZdZee d<dZe e d<dZee d<dZee d<dZee d<dZe e d<dZeee d<dZe e d<dZ!e e d<dZ"e e d<dS) LowerSettinga' Basic configuration for lowering stack. Args: input_specs: Specs for inputs to engine, can either be a single size or a range defined by Min, Optimal, Max sizes. explicit_batch_dimension: Use explicit batch dimension during lowering. explicit_precision: Use explicit precision during lowering. max_workspace_size: The maximum workspace size. The maximum GPU temporary memory which the TensorRT engine can use at execution time. strict_type_constraints: Require TensorRT engine to strictly follow data type setting at execution time. customized_fuse_pass: List of custmozied pass to apply during lowering process. lower_basic_fuse_pass: Enable basic pass fuse duirng lowering, i.e. fuse multiple operations as (a->b->c->d)=>(e). Current basic fuse patterns are: permute->linear permute->matmul verbose_log: Enable TensorRT engine verbose log mode. algo_selector: Enable TensorRT algorithm selector at execution time. timing_cache_prefix: TensorRT timing cache file path. TensorRT engine will use timing cache file at execution time if valid timing cache file is provided. save_timing_cache: Save updated timing cache data into timing cache file if the timing cache file is provided. cuda_graph_batch_size (int): Cuda graph batch size, default to be -1. preset_lowerer (str): when specified, use a preset logic to build the instance of Lowerer. only used by explicit batch dim with dynamic shape mode. In general, we use 2 GPU setting with 2 stream on each. Set total number to 8 as a safe default value. dynamic_batch: enable the dynamic shape in TRT with dim=-1 for the 1st dimension. tactic_sources: tactic sources for TensorRT kernel selection. Default to None, meaning all possible tactic sources. correctness_atol: absolute tolerance for correctness check correctness_rtol: relative tolerance for correctness check use_experimental_rt: Uses the next generation TRTModule which supports both Python and TorchScript based execution (including in C++). )default_factory input_specsTexplicit_batch_dimensionFexplicit_precisioni@max_workspace_sizestrict_type_constraintscCs tgSN)rbuild_from_passlistrrrr Us zLowerSetting.customized_fuse_passcCstttgSr()rr)r r rrrr r*Xslower_basic_fuse_pass verbose_logNtiming_cache_prefixsave_timing_cachecuda_graph_batch_sizepreset_lowereropt_profile_replica dynamic_batchtactic_sourcesg?correctness_atolcorrectness_rtoluse_experimental_rt)#rrrrdcfieldlistr#rr rr$rr%r&rr'r+rr,r- algo_selectorr/strr0r2r3r5r6r7rr8floatr9r:rrrr r!*s2 #             r!) dataclassesr;typingrrrrtorchrtorch.fx.passes.pass_managerrinput_tensor_specr passes.lower_basic_passr r utilsr dataclassr r!rrrr s