o Ơi{@sUddlmZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlZddlZddlZddlZddlZddlmZddlm Z ddl!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6e7e8Z9dZ:dZ;dZGddde Z?ej@e?jAej@e?jBej@e?jCej@iejDe?jAejDe?jBejDe?jCejDiejEe?jAejEe?jBejEe?jCejEiejFe?jAejFe?jBejFe?jCejFiejGe?jAejGe?jBejGe?jCejGiejHe?jAeHe?jBejHe?jCejHiiZIdeJd<e*dr/e?jAejKe?jBejHe?jCejHieIejH<dd"d#ZLddd*d+ZMddd.d/ZNdd4d5ZOdd8d9ZP :dddBdCZQ :dddGdHZRddIdJZSddNdOZT PdddSdTZUdUdVfddZd[ZVdd]d^ZWddbdcZX :dddgdhZYddidjZZddodpZ[ddrdsZ\ddudvZ]ddxdyZ^dd|d}Z_ddddZ` ddddZaeTmoduletorch.fx.GraphModule delete_moduler7NonecCs&|t|r~tjtdS)z This is a helper function to delete the instance of module. We first move it to CPU and then delete the object. This function ensures the GPU memory occupied by the module is released effectively after this call N)r, CPU_DEVICEr cuda empty_cachegccollect)r?rAr(r(r)deallocate_modules   rHuse_python_runtimeOptional[bool]cCsn|}d}|dur d}nz ddlm}d}d}Wn ty$d}d }Ynwtd |r,d nd d |d|S)zParses a user-provided input argument regarding Python runtime Automatically handles cases where the user has not specified a runtime (None) Returns True if the Python runtime should be used, False if the C++ runtime should be used Nzas requested by userr)TorchTensorRTModuleFz,since C++ dependency was detected as presentTz1since import failed, C++ dependency not installedzUsing z Python-onlyDefaultz Torch-TRT Runtime ())torch_tensorrt.dynamo.runtimerL ImportErrorloggerinfo)rIusing_python_runtimereasonrLr(r(r)use_python_runtime_parsers   rU gt_tensor torch.Tensor pred_tensorfloatcCs~|tj}|tj}t|dkst|dkr*tj||ddddr*dStjjj||ddd}| }|S) Ngg-C6?T)atolrtol equal_nang?rgư>)dimeps) flattenr,r r;sumallclosenn functionalcosine_similarityrdetachitem)rVrXres_tresr(r(r)rdsrdinputs0Sequence[Union[Input, torch.Tensor, FakeTensor]]cCs^|D]*}t|tjrt|jrdSqt|tr#|jtjjkr"dSqt dt |ddS)zs Return true if any of inputs have dynamic shapes. Supported types are torch_tensorrt.Input | torch.Tensor TzInvalid input type (z@) found. Supported types are torch_tensorrt.Input | torch.TensorF) isinstancer Tensorcontains_sym_intshaper shape_mode _ShapeModeDYNAMICAssertionErrortype)riinputr(r(r)input_is_dynamics   rurKrtrdevice torch.devicemodestrUnion[int, torch.Tensor]cCs<|jr |jddSt|dkr|||S|j|S)N opt_shaper)is_shape_tensorrnlenexample_tensorr, torch_tensor)rtrvrxr(r(r)get_torch_tensors   r Sequence[Input] | Dict[str, Any] Union[Device, torch.device, str]HSequence[Union[int, torch.Tensor]] | Dict[str, Union[int, torch.Tensor]]cCst|}t|tr3i}|D]!\}}t|tttfr#t||||<qt|tr0t|||||<q|Sg}|D])}t|trH| t|||q7t|t j rW| | |q7t dt|d|S)z Return the torch_tensor from the Input object. If mode is set, this implies user is using dynamic shaped inputs and return the corresponding input based on the mode requested. z Input type z is not a valid type)to_torch_devicerkdictitemslisttupleget_torch_inputsrrappendr rlr,rrrs)rirvrx result_dictkv result_listrtr(r(r)rs$    rcCstd}t|D]}t|tjjjtjfr|jSqt| D] }t|tjr.|jSq!|dur8t t }|S)zB Returns the device on which the module parameters exist. N) r parametersrkr rb parameter Parameterrlrvbuffersrr)r?rvrbufferr(r(r)get_model_devices    r parent_loggerrlevelcCs|r||tjrS|tjkrtjjj}n3|tj kr tjjj }n(|tj kr+tjjj }n|tj kr6tjjj }n|tj krAtjjj }nt|dtjjt|dSdS)z Sets the log level to the user provided level. This is used to set debug logging at a global level at entry points of tracing, dynamo and torch_compile compilation. And set log level for c++ torch trt logger if runtime is available. z is not valid log levelN)setLevelrtorch_tensorrt_runtimeloggingDEBUGr!ILoggerSeverityVERBOSEINFOWARNINGERRORCRITICALINTERNAL_ERRORrrr opstensorrtset_logging_levelr1)rr log_levelr(r(r) set_log_levels            rF5Input | torch.Tensor | Sequence[Any] | Dict[Any, Any]disable_memory_format_checkcCs@t|dur WddSt|tr|WdSt|tjtttfr:tjt ||dWdSt|t t frfg}|D] }t ||d}| |qEt|t rZ|nt | WdSt|trt}|D]\}}t ||d}|||<qr|WdStdt|dd1swYdS)zc We take a nested group of torch.Tensors or scalars and convert them into torchtrt.Input's N)rInvalid input type z2 encountered in the dynamo_compile input parsing. LAllowed input types: {torch_tensorrt.Input, torch.Tensor, list, tuple, dict})rrkrr rlr1rYr7 from_tensortensorrrprepare_inputsrrr ValueErrorrs)rirtorchtrt_input_list input_objtorchtrt_inputtorchtrt_inputs_dictkeyr(r(r)r,sP      'rcCs|SNr()xr(r(r)esrattribute_to_extractapply_fnCallable[[Any], Any]c Cst|tjtfr|t||dSt|tttfr%t|}|t||dSt|t t frIg}|D] }t |||}| |q0t|t rE|St |St|t rft }|D]\}}t |||}|||<qU|Stdt|dd)zParses complex structures of Tensors and returns a mirrored structure Extracts key attributes of each singular element, while reconstructing the struct Optionally applies a function to each attribute before returning Nrz* encountered during Dynamo input parsing. r)rkr rlrgetattrr1rYr7rrrparse_complex_tensor_structsrrrrrs) rirr inputs_torchrrrrrr(r(r)rbs<     rrcCstdd|DS)z> Returns true if the given tensor has symbolic shape. css|] }t|tjVqdSr)rkr SymInt).0r]r(r(r) sz#contains_sym_int..)anyrr(r(r)rmsrmsymbolic_integer torch.SymIntDict[str, int]c Cs|j}|j}|j}|j|dp||}|j|dp(|j|dp(||j}|s/J|t |j t |j }}|dkr@dn|}i}||d<||d<t |t jjjt fr\t ||d<|S)zO This function returns the min, max, opt values of a symbolic integer. Nrminmaxopt)nodeexpr shape_env var_to_rangeget bound_sympy var_to_valunbacked_var_to_valxreplacer1lowerupperrksympycorenumbersInteger) rrrr var_rangevar_valmin_valmax_val min_max_optr(r(r)extract_var_range_infos$    r-Union[torch.Tensor, FakeTensor, torch.SymInt] Optional[str]%Sequence[Union[int, Tuple[int, int]]]cCsg}t|tr||t |St|tjr6t|}|r'|||t |S||d|dft |St|tjrE|dt |St|tjtfr\|j D] }| t ||dqPt |S)a8 This is a helper function used to print/return the shape of the tensor. For regular torch.tensor's, it returns the static shape. For symbolic tensors, eg:(1, s0, 4), this function returns [1, [min, max], 4]. The min and max correspond to the lower and upper values of s0 symbolic dimension. rrr)rx) rkr1rr rrSymFloatrlrrnextendunwrap_tensor_shaper)rrx tensor_shaper dimensionr(r(r)rs$      rcCstt|tjtfr |jSt|tttfrt|jSt|tj r"tj St|tj r+tj S|dur1dSt dt|)za Returns the dtype of torch.tensor or FakeTensor. For symbolic integers, we return int64 NzFound invalid tensor type )rkr rlrrr1rYr7rrr9rr;rrsrr(r(r)unwrap_tensor_dtypes   rio_nodesSequence[torch.fx.Node] attr_type Sequence[Any]cCsx|dvsJ|dkr tnt}g}|D]'}d|jvr9|jd}t|ttfr2|D] }|||q'q|||q|S)zJ Returns a list of attributes (shapes or dtypes) of the I/O nodes )rnrrnval)rrmetarkrrr)rrattr_fngraph_io_attrsrmetadatarr(r(r)get_graph_io_attrss   rdryrun_trackerc Csdd|jjD}t|d}t|d}||_||_dd|jjD}g}|D]}||jq&t|d}t|d} ||_| |_dS)zb Parse the graph I/O shape/dtype info for the whole graph and store in the dryrun tracker cSg|] }|jdkr|qS) placeholderoprrr(r(r) z"parse_graph_io..rnrcSroutputrrr(r(r)rrN) graphnodesr input_shapes input_dtypesrall_input_nodes output_shapes output_dtypes) r?r input_nodesrrmark_output_nodes output_nodesrrrr(r(r)parse_graph_ios     r*Optional[Union[Device, torch.device, str]]cCsHt|tr |tjSt|tjr|S|durttjSt|S)zSCast a device-type to torch.device Returns the corresponding torch.device N)rkrr,r rvrDcurrent_devicervr(r(r)rs    rrcCs t|S)zeCast a device-type to torch_tensorrt.Device Returns the corresponding torch_tensorrt.Device )r_fromr r(r(r)to_torch_tensorrt_device$s r kwargs5Tuple[CompilationSettings, Optional[BaseEngineCache]]cst}|rSd|vrt|dkr|d}d|vr7d|vr&|dtjur&td|d|d<tjdtdd|d=d d t|Dfd d | D}t |fi|}d |vrudd |d D}t|dkrrt dtj tj}||_t|j|_t|j|_d|vrt d|jjd|jrt dd|_d}|ds|dr|ddur|d}nddlm}|dtj}|dtj}|||}|dr|d|_t d|||fS)zParses the kwargs field of a Dynamo backend Args: kwargs: Keyword arguments dictionary provided to the backend Returns: CompilationSettings object with relevant kwargs optionsrtruncate_long_and_doubletruncate_doublez~Provided configuration for "truncate_double" and deprecated API "truncate_long_and_double". Please only use "truncate_double".zCompiler option "truncate_long_and_double" is deprecated in favor of "truncate_double" as int64 is now natively supported. This option will be removed in the next version.r) stacklevelcSsh|]}|jqSr()name)rattrr(r(r) Xz&parse_dynamo_kwargs..csi|] \}}|vr||qSr(r()rrr valid_attrsr(r) Ysz'parse_dynamo_kwargs..enabled_precisionscSsh|]}t|qSr()rr )rer(r(r)r^srz&No precision specified, defaulting to rvz@Device not specified, using Torch default current device - cuda:zO. If this is incorrect, please specify an input device, via the device keyword.zkDetected require_full_compilation=True for a torch.compile run. This option has no effect in torch.compile.FNcache_built_enginesreuse_cached_enginescustom_engine_cache)DiskEngineCacheengine_cache_direngine_cache_sizetorch_executed_opszCompilation Settings: %s )rr}rTRUNCATE_DOUBLErwarningswarnDeprecationWarningrrrrQrRENABLED_PRECISIONENABLED_PRECISIONSrrUrIr rvgpu_idrequire_full_compilationwarningr#torch_tensorrt.dynamo._engine_cacherENGINE_CACHE_DIRENGINE_CACHE_SIZEr!)r settings valid_kwargsr engine_cacherrr r(rr)parse_dynamo_kwargs.sr            r12.devmin_torch_versionCallable[..., Any]csdfdd }|S)ak Create a decorator which verifies the Torch version installed against a specified version range Args: min_torch_version (str): The minimum required Torch version for the decorated function to work properly Returns: A decorator which raises a descriptive error message if an unsupported Torch version is used fr4r-csdfdd }|S)Nargsrr r-csNt}ttj}||kr tddddtj|i|S)NzExpected Torch version z or greater, z when calling z. Detected version )rparser r2rr)r6r  min_versioncurrent_version)r5r3r(r)function_wrappers   zEreq_torch_version..nested_decorator..function_wrapper)r6rr rr-rr()r5r:r3)r5r)nested_decorators z+req_torch_version..nested_decoratorN)r5r4r-r4r()r3r<r(r;r)req_torch_versionsr= new_modulerefitted_module arg_inputs kwarg_inputscCs@||||i|}}t|t|krtddSt||S)Nz8The output types are different. Output check is skipped.T)rsrQr*check_output_equal)r>r?r@rA old_outputs new_outputsr(r(r)check_module_outputs   rEoutput1output2r[rZcCst|t|krtddSt|tjr%|j|jkrdSt||||St|tt frKt |t |kr6dSt ||D]\}}t ||sGdSdSn't|t rr||krZdSt ||D] \}}t ||sodSqcdStddS)NzLThe output types are different. Check_output_equal will always return false.FTz\The output type is not supported to be checked. Check_output_equal will always return false.)rsrQr*rkr rlrnrarrr}ziprBrkeysvalues)rFrGr[rZabr(r(r)rBs:     rBexported_programtorch.export.ExportedProgramr6 list[Any]dict[str, Any]tuple[Any, Any]c Csdddlmm}ddlm}|jj}|dur|||}|||f\}}tdd|D}||fS)aFlatten args, kwargs using pytree, then, check specs. Args: args: List[Any] original args passed to __call__ kwargs: Dict[str, Any] original kwargs passed to __call Returns: A tuple of (flat_args, received_spec) flat_args is flattend args / kwargs received_spec is the pytree spec produced while flattening the tuple (args, kwargs) rN)reorder_kwargscss|]}|dVqdS)rNr()rrr(r(r)rsz+get_flat_args_with_check..) torch.utils._pytreeutils_pytreetorch.export._tree_utilsrR call_specin_spectree_flatten_with_pathr) rMr6r pytreerRrXflat_args_with_path received_spec flat_argsr(r(r)get_flat_args_with_checks  r^gm target_optorch._ops.OpOverload List[Any]csfdd|jjDS)a Return the list which has the metadata of all the target_op nodes present in the graph. csg|] }|jkr|jqSr()targetrrr`r(r)rsz get_metadata..)rr)r_r`r(rer) get_metadata srfrcsJfdd|jjD}t|t|ksJt|D] \}}|||_qdS)rccsg|] }|jkr|qSr()rdrrer(r)rrz set_metadata..N)rrr} enumerater)r_r`r target_nodesidxrr(rer) set_metadatas  rjmatch_and_replacementscCsB|D]}|j|j}t|jdksJd|jd}|j|_qdS)z Copy the metadata from anchor node to the replacement node. This should be used if the anchor node is replaced with only a single replacement node i.e one-one replacement. rz3Found more than 1 replacements for the anchor node.rN) nodes_mapanchorr} replacementsr)rkmatch_and_replacement anchor_nodereplacement_noder(r(r) copy_metadatas   rrrList[torch.fx.node.Node]cCs^g}t|tjjjr|||St|ttfr&|D] }|t |q|St dt |)Nzhexpect torch.fx.node.Node or a tuple/list of torch.fx.node.Node type, got unexpected types: type(nodes)=) rkr fxrNoderrrr flatten_nodesrrs)rretrr(r(r)rv-s  rvcCsRdd|jjD}t|dksJ|dj}t|}t|dks"Jdd|DS)NcSrrrrr(r(r)r>rz'get_output_metadata..rcSsg|]}|jqSr()rrr(r(r)rCr)rrr}r6rv)r_outputsrr(r(r)get_output_metadata;s  ryrr List[dtype]cCsg}t|tjjjrcd|jvrE|jd}t|ttjfr7|r,|jtj kr,| tj |S| t |j|St|tj rC| tj|Sd|jvrZ|jd}| t |j|Std|jdt|ttfrx|D] }|t|ql|Stdt|d)Nr tensor_metaz node.name=zF: metadata does not exist, expect metadata exists for each output nodezgot unexpected type zM, expected type is a torch.fx.node.Node or a tuple/list of torch.fx.node.Node)rkr rtrrurrrlrfloat64rr;r rr9rrrrrget_output_dtypesrs)rrr output_metaeler(r(r)r}Fs6       r}cCstjddS)Ni)psutilProcess memory_inforssr(r(r(r)get_cpu_memory_usagecsrcCsttjrtjtjtjtjt dkrSt j dddkrUzt d}|ddkrAtdWdSWdStyRtdYdSwdSdS) NLinux#TORCHTRT_ENABLE_BUILDER_MALLOC_TRIM01z libc.so.6rrzFailed to release CPU memory.)rFrGr rD is_available synchronizerE ipc_collectplatformsystemosenvironrctypesCDLL malloc_trimrQr* Exception)libcr(r(r)release_host_and_device_memorygs$        r)rr+r,rr-r.)T)r?r@rAr7r-rBr)rIrJr-r7)rVrWrXrWr-rY)rirjr-r7)rK)rtrrvrwrxryr-rz)rirrvrrxryr-r)r?r@r-rw)rrrrr-rB)F)rirrr7r-r)rirrryrrr-r)rrWr-r7)rrr-r)rrrxrr-r)rrr-r)rrrryr-r)r?r@rrr-rB)rvrr-rw)rvrr-r)r rr-r )r2)r3ryr-r4) r>r@r?r@r@rrArr-r7) rFrrGrr[rYrZrYr-r7)rMrNr6rOr rPr-rQ)r_r@r`rar-rb)r_r@r`rarrbr-rB)rkrbr-rB)rrr-rs)r_r@r-rb)rrrr7r-rz)r-r)r-rB)l __future__rrrFrrrr# dataclassesrrenumrtypingrrrr r r r r rr4rrrr!r torch._subclasses.fake_tensorr"torch.fx.experimental.proxy_tensorrtorch_tensorrt._Devicertorch_tensorrt._enumsrtorch_tensorrt._featuresrtorch_tensorrt._Inputrtorch_tensorrt._utilsrtorch_tensorrt.dynamortorch_tensorrt.dynamo._defaultsrr+rtorch_tensorrt.dynamo._settingsr packagingrtypesr getLoggerr"rQCOSINE_THRESHOLD DYNAMIC_DIMRTOLATOLrCrr5r%r&r'r8r9r:r;r7r*__annotations__r6r>rHrUrdrurrrrrrrmrrrrrrr r1r=rErBr^rfrjrrrvryr}rrr(r(r(r)s (                 #       9 0       i%  '