o ƠiG@s\ddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZmZddlmZddlmZdd lmZdd lmZejrzddl Zdd l!m"Z"dd l#m$Z%dd l&m'Z'eeej(e"fZ)neeej(fZ)ej*rddl+Z,dd l-m$Z.ddl-m/Z0ej1rddl2m3Z3dd l4m$Z5ddl4m6Z7ddl4m8Z9ddl4m:Z;ddl4mm?Z@eAeBZCgdZDdTddZEGdddeZFGdddeZGdUd"d#ZHdVd'd(ZI )    dWdXd3d4Z$e    dYdZd9d:Z8d[d;d<ZJ =    ) d\d]d@dAZ/d^d_dCdDZ:d^d_dEdFZK Bd^dGddddHdIdJd`dRdSZLdS)a) annotationsN)Enum)AnyCallableListOptionalSequenceSetUnion)dtype)ENABLED_FEATURESneeds_cross_compile)Input) _defaults)CudaGraphsTorchTensorRTModule) TypeGuard)InputTensorSpec)compile)LowerPrecision)convert_method_to_trt_engine)ExportedProgram)1convert_exported_program_to_serialized_trt_engine)cross_compile_for_windows)$load_cross_compiled_exported_program)$save_cross_compiled_exported_program)trace)rrrrsaveloadinputsSequence[Input | torch.Tensor]return%TypeGuard[List[Input | torch.Tensor]]cCtdd|DS)Ncs |] }t|tjtfVqdSN) isinstancetorchTensorr.0ir+K/home/ubuntu/.local/lib/python3.10/site-packages/torch_tensorrt/_compile.py Ez*_non_fx_input_interface..allrr+r+r,_non_fx_input_interfaceBr2c@s$eZdZdZdZdZdZdZdZdS)_IRTypez?Enum to determine the type of IR selected for model compilationrN) __name__ __module__ __qualname____doc__tsfxdynamo torch_compileexported_programr+r+r+r,r4Hsr4c@s eZdZdZdZdZdZdZdS) _ModuleTypez5Enum to determine the type of model provided as inputrr5r6r7N)r9r:r;r<nnr=r>epr+r+r+r,rBRs rBmodulercshtfddtjjtjjfDrtjSttjj rtjStt r&tj Sttj j r0tj Std)Nc3s|]}t|VqdSr$)r%r)trEr+r,r-\s  z%_parse_module_type..zModule is an unknown format)anyr&jit ScriptModuleScriptFunctionrBr=r%r> GraphModulerrDrCModule RuntimeErrorrHr+rHr,_parse_module_type[s  rP module_typeirstrc s~tfddtjtjfD}tfddtjtjfD}tjk}tfdddD}dk}dk}dk}|rH|rHtjrDtjSt d |r^|r^t j d t d d tj rZtjSt d |sb|rn|rntjrjtjSt d|r||r|tjrxtjSt ddkrtjr|rtdtjStjr|rtjrtdtjStjr|rtdtjSt dtjdtjdtj dt d)Nc3|]}|kVqdSr$r+rFrQr+r,r-lz!_get_target_fe..c3rTr$r+rFrUr+r,r-mrVc3rTr$r+)r)opt)rRr+r,r-prV) torchscriptr=r>r?r@zdRequested using the TS frontend but the TS frontend is not available in this build of Torch-TensorRTBFX frontend is deprecated. Please use the Dynamo frontend instead.r6 stacklevelzdRequested using the FX frontend but the FX frontend is not available in this build of Torch-TensorRTzlRequested using the Dynamo frontend but the Dynamo frontend is not available in this build of Torch-TensorRTzRequested using the Torch-TensorRT torch.compile backend but the Torch-TensorRT torch.compile backend is not available in this build of Torch-TensorRTdefaultz,ir was set to default, using dynamo frontendzInput is a torchscript module but the ir was not specified (default=dynamo), please set ir=torchscript to suppress the warning.zLModule was provided in an unsupported format Installed frontends: Dynamo - z TorchScript - z FX - )zUnknown ir was requested)rIrBrCr=r>rDr torchscript_frontendr4 ValueErrorwarningswarnDeprecationWarning fx_frontenddynamo_frontendr?r@loggerinfowarning) rQrRmodule_is_tsablemodule_is_fxablemodule_is_exportableir_targets_torchscript ir_targets_fxir_targets_dynamoir_targets_torch_compiler+)rRrQr,_get_target_feksj       ror\Optional[Sequence[InputType]] arg_inputs!Optional[Sequence[Sequence[Any]]] kwarg_inputsOptional[dict[Any, Any]]enabled_precisions(Optional[Set[Union[torch.dtype, dtype]]]kwargsTtorch.nn.Module | torch.jit.ScriptModule | torch.fx.GraphModule | Callable[..., Any]cKs |dur|ng}|dur|ntj}t|} t| |} | tjkrD|} | tjkr1t dt j |} t |s7Jt| f||d|} | S| tjkrtjdtddtjsXtdt j|vsctjj|vrgtj} nt j|vsrtjj|vrvtj} ntd|d ddd}||sJt ||f| ddd|}|S| tj!kr|dur|durt"d|dur|durt"d|dur|}|duri}ddl#m$}t%|t&j'j(s|g}||}||}t)||fd|i|}t*|f||d|}|S| tj+krt+|fd|i|Std)aCompile a PyTorch module for NVIDIA GPUs using TensorRT Takes a existing PyTorch module and a set of settings to configure the compiler and using the path specified in ``ir`` lower and compile the module to TensorRT returning a PyTorch Module back Converts specifically the forward method of a Module Arguments: module (Union(torch.nn.Module,torch.jit.ScriptModule): Source module Keyword Arguments: inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. :: inputs=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] arg_inputs (Tuple[Any, ...]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (dict[Any, ...]): Optional, kwarg inputs to the module forward function. enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels ir (str): The requested strategy to compile. (Options: default - Let Torch-TensorRT decide, ts - TorchScript with scripting path) **kwargs: Additional settings for the specific requested strategy (See submodules for more info) Returns: torch.nn.Module: Compiled Module, when run it will execute via TensorRT NModule was provided as a torch.nn.Module, trying to script the module with torch.jit.script. In the event of a failure please preconvert your module to TorchScript)rrurYr6rZz._fx_input_interface..r/r1r+r+r,_fx_input_interface r3z$compile.._fx_input_interfaceTF)lower_precisionexplicit_batch_dimension dynamic_batch2'arg_inputs' and 'inputs' should not both be None.>'arg_inputs' and 'inputs' should not be used at the same time.rprepare_inputsrsrqruru:Module is an unknown format or the ir requested is unknown)rrzr r{),rENABLED_PRECISIONSrPror4r=rBrCrerfr&rJscriptr2torchscript_compiler>r`rarbr rcrOfloat16torch_tensorrtr halfrFP16float32floatFP32r_ fx_compiler?AssertionErrortorch_tensorrt.dynamo.utilsrr% collectionsabcr dynamo_tracedynamo_compiler@)rErRrrqrsrurw input_listenabled_precisions_setrQ target_irts_modcompiled_ts_moduler}r|compiled_fx_modulertorchtrt_arg_inputstorchtrt_kwarg_inputs exp_programtrt_graph_moduler+r+r,rs0               rtorch.nn.Module file_path(Optional[Sequence[Input | torch.Tensor]]Nonec Kstdkstddkrtdtdtd|s$td|dur*|ntj}|dur9|dur9td|durE|durEtd |pH|}|durOi}dd lm }t |t j j s_|g}||} ||} t|| fd | i|} td t| f| |d |} t| |tddS)aCompile a PyTorch module using TensorRT in Linux for Inference in Windows Takes an existing PyTorch module and a set of settings to configure the compiler and it will convert methods to AOT graphs which call equivalent TensorRT serialized engine info into the disk in the specified file_path user provided. It will then allow user to load the deserialized model from the disk in Windows. Note: the model cross compiled for windows in Linux environmen can only be loaded in Windows. Argument: module (torch.nn.Module): Source module file_path (str): the file path to store the serialized module into the disk Keyword Arguments: inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. :: inputs=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] arg_inputs (Tuple[Any, ...]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (dict[Any, ...]): Optional, kwarg inputs to the module forward function. enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels **kwargs: Additional settings for the specific requested strategy (See submodules for more info) Linuxr64bitznCross compile for windows is only supported on x86-64 Linux architecture, current platform: platform.system()=z, platform.architecture()[0]=;File path cannot be empty. Please provide a valid file pathNrrrrsz successfully exported the modulerz6successfully compiled and saved the module for windows)platformsystem architecturerOr_rrrrrr%rrrrredebug dynamo_cross_compile_for_windows+dynamo_save_cross_compiled_exported_program) rErrrqrsrurwrrrrrtrt_gmr+r+r,rFsT.   rcKs&ddlm}tj||di|d}|S)z Returns a boxed model which is the output of torch.compile. This does not compile the model to TRT. Execute this model on sample inputs to compile the model to TRT. r)torch_tensorrt_backendF)backenddynamicoptions)torch_tensorrt.dynamo.backendrr&r)rErwrboxed_fnr+r+r,r@s  r@forward method_namebytescKsP|dur|ntjh}|dur|durtd|dur"|dur"td|p%|}t|} t| |} | tjkrT|} | tjkrFt dtj |} t | f|||d|} | S| tjkr]td| tjkr|durhi}ddlm} t|tjjsx|g}| |}| |}tjj||fd |i|}t|ft|||d |S| tjkrtd td ) aConvert a TorchScript module method to a serialized TensorRT engine Converts a specified method of a module to a serialized TensorRT engine given a dictionary of conversion settings Arguments: module (Union(torch.nn.Module,torch.jit.ScriptModule): Source module Keyword Arguments: inputs (List[Union(torch_tensorrt.Input, torch.Tensor)]): **Required** List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type. :: input=[ torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1 torch_tensorrt.Input( min_shape=(1, 224, 224, 3), opt_shape=(1, 512, 512, 3), max_shape=(1, 1024, 1024, 3), dtype=torch.int32 format=torch.channel_last ), # Dynamic input shape for input #2 torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings ] arg_inputs (Tuple[Any, ...]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (dict[Any, ...]): Optional, kwarg inputs to the module forward function. enabled_precision (Set(Union(torch.dtype, torch_tensorrt.dtype))): The set of datatypes that TensorRT can use when selecting kernels ir (str): The requested strategy to compile. (Options: default - Let Torch-TensorRT decide, ts - TorchScript with scripting path) **kwargs: Additional settings for the specific requested strategy (See submodules for more info) Returns: bytes: Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs Nrrry)rrruzrOr?rrr%rrrrr8dynamo_convert_exported_program_to_serialized_trt_enginetupler@)rErrrqrsrRrurwrrQrrserialized_enginerrrrr+r+r,rsz+        rcCst|S)a) Load an ExportedProgram file in Windows which was previously cross compiled in Linux Arguments: file_path (str): Path to file on the disk Raises: ValueError: If the api is not called in windows or there is no file or the file is not a valid ExportedProgram file )+dynamo_load_cross_compiled_exported_program)rr+r+r,r's rcCsztd|dtj|}|WSty&tjd|dddYnwztd|dtj|}|WStyStjd|dddtd|d w) aj Load either a Torchscript model or ExportedProgram. Loads a TorchScript or ExportedProgram file from disk. File type will be detect the type using try, except. Arguments: file_path (str): Path to file on the disk Raises: ValueError: If there is no file or the file is not either a TorchScript file or ExportedProgram file zLoading the provided file z using torch.jit.load()z5 via torch.jit.load() failed with the following errorT)exc_infoz using torch.export.load()z8 via torch.export.load() failed with the following errorz The file zb doesn't correspond to a valid Torchscript module or ExportedProgram. Please verify the file path.) rerr&rJr Exceptionrfexportr_)r ts_modulerr+r+r,r4s0        rrATr6) output_formatrrqrsretracepickle_protocolr Optional[Sequence[torch.Tensor]]Optional[dict[str, Any]]rboolrintc st|tr|j}t|} hd} |dur!tdd|Ds!td|r)|r)td|p,|}|dur3i}|rDtdd|DrDtd| vrPtd d d krdt d krdtd t d|sjtd| t j krstd| t j krtfdddDstd|durtdtj||dS| t jkrۈdkrtd|durtddkrtjj|||ddSd kri} d|vr|d} tjj|| |ddStd| t jkrdkrtjj|||d} tj| |dS|s=ddlm} |dur td | |}dkrtjj|||ddSd kr9i} d|vr.|d} tjj|| |ddStd|durFtd!tjj|t||d"d#}dkrbtjj|||ddSd kr}i} d|vrr|d} tjj|| |ddStddS)$a Save the model to disk in the specified output format. Arguments: module (Optional(torch.jit.ScriptModule | torch.export.ExportedProgram | torch.fx.GraphModule | CudaGraphsTorchTensorRTModule)): Compiled Torch-TensorRT module inputs (torch.Tensor): Torch input tensors arg_inputs (Tuple[Any, ...]): Same as inputs. Alias for better understanding with kwarg_inputs. kwarg_inputs (dict[Any, ...]): Optional, kwarg inputs to the module forward function. output_format (str): Format to save the model. Options include exported_program | torchscript | aot_inductor. retrace (bool): When the module type is a fx.GraphModule, this option re-exports the graph using torch.export.export(strict=False) to save it. This flag is experimental for now. pickle_protocol (int): The pickle protocol to use to save the model. Default is 2. Increase this to 4 or higher for large models >rX aot_inductorrANcss|] }t|tjVqdSr$)r%r&r')r)inputr+r+r,r-vs  zsave..zQNot all inputs provided are torch.tensors. Please provide torch.tensors as inputsrcss|]}|duVqdSr$r+)r)valuer+r+r,r-rVzkwargs should not include None.zProvided output_format zG is not supported. Supported options are exported_program | torchscriptrrz4The AOT Inductor format is only supported on Linux, z, is not a supported platform for this formatrzInput model is of type nn.Module. Saving nn.Module directly is not supported. Supported model types torch.jit.ScriptModule | torch.fx.GraphModule | torch.export.ExportedProgram.c3rTr$r+)r)frr+r,r-rV)rArzProvided model is a torch.jit.ScriptModule but the output_format specified is not torchscript. Other output formats are not supportedz^Provided model is a torch.jit.ScriptModule, inputs or arg_inputs is not necessary during save.rXzProvided model is a torch.export.ExportedProgram but the output_format specified is torchscript. Please verify the output_formatzProvided model is a torch.export.ExportedProgram, inputs or arg_inputs is not necessary during save, it uses the inputs or arg_inputs provided during export and compilerA)rinductor_configs)r package_pathzAttempted to serialize an exported program with an unsupported format. Exported programs support exported_program and aot_inductor)example_kwarg_inputsr)rzqProvided model is a torch.fx.GraphModule and retrace is False, inputs or arg_inputs is not necessary during save.zProvided model is a torch.fx.GraphModule and retrace is True, however the inputs or arg_inputs are empty. Please provide valid torch.tensors as inputs or arg_inputs to trace and save the modelF)rwstrict)r%rcompiled_modulerPr0r_rrIvaluesrrrBrCr=rergr&rJrrDr _inductoraoti_compile_and_packagerOr>rtorch_tensorrt.dynamo._exporterr)rErrrrqrsrrrwrQaccepted_formatsr module_tsrrr+rr,rYs                  r)rrr r!)rErr rB)rQrBrRrSr r4)r\NNNN)rErrRrSrrprqrrrsrtrurvrwrr rx)NNNN)rErrrSrrrqrrrsrtrurvrwrr r)rErrwrr r)rNNNr\N)rErrrSrrrqrrrsrtrRrSrurvrwrr r)r)rrSr r)rErrrSrrSrrrqrrsrrrrrrwrr r)M __future__rcollections.abcrloggingrr`enumrtypingrrrrrr r r&torch_tensorrt._enumsr torch_tensorrt._featuresr r torch_tensorrt._Inputrtorch_tensorrt.dynamors  $                    D  c m '