o }oi3p@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZmZmZmZmZmZddlZddlmZmZdd lmZed \ZZerldd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$ed \Z%Z&ed\Z'Z(ed\Z)Z(e*Z+ddZ,dd?d:d;Z8dS)@) annotationsN) OrderedDict) getLogger)Path) MethodType)AnyDictListSequenceTupleUnion)add_casts_around_normsreplace_for_export) safe_import polygraphy)bytes_from_path) CreateConfigProfileengine_bytes_from_networkengine_from_bytesnetwork_from_onnx_pathtensorrttorch_tensorrtz cuda.cudartcCs<tjtjtjtjtjtjtjtjtjtjtjtjtjtjiS)z) Map of TRT dtype -> Torch dtype ) trtint32torchfloat32float16bfloat16int64int8boolr"r"V/home/ubuntu/.local/lib/python3.10/site-packages/nemo/export/tensorrt_lazy_compiler.pytrt_to_torch_dtype_dict/sr$ input_shape Sequence[int]dynamic_batchsizeSequence[int] | NonecCsPd dd}|r|||d}|||d}|||d }n|}}}|||fS) zb Given a sample input shape, calculate min/opt/max shapes according to dynamic_batchsize. r%r& scale_numintcSsg|}||d<|S)Nrr")r%r) scale_shaper"r"r#scale_batch_sizeCsz,get_profile_shapes..scale_batch_sizerN)r%r&r)r*r")r%r'r,min_input_shapeopt_input_shapemax_input_shaper"r"r#get_profile_shapes>s   r2cCs|i}|s|S|D]3}|D].}g}||}tt|dD]}|d||d|kr/||qt|dkr:|||<q q|S)z This method calculates dynamic_axes to use in onnx.export(). Args: profiles: [[min,opt,max],...] list of profile dimensions rr.)rangelenappend)profiles dynamic_axesprofilekeyaxesvalsir"r"r#get_dynamic_axesRs   r=cCs6|d}|dkrtd|t|dkr|dSdS)z[ Error reporting method for CUDA calls. Args: cuda_ret: CUDA return code. rz CUDA ERROR: r-N) RuntimeErrorr4)cuda_reterrr"r"r#cuassertgs  rAc@seZdZdZdS) ShapeErrorzM Exception class to report errors from setting TRT plan input shapes N)__name__ __module__ __qualname____doc__r"r"r"r#rBusrBc@s4eZdZdZd ddZddZddZd d d ZdS) TRTEnginezK An auxiliary class to implement running of TRT optimized engines NcCs||_|ptd|_|jd|jtt|j|_t|_d|_ |j |_ g|_ g|_ g|_d|_i|_t}t|jjD]6}|j|}|j|tjjkrY|j |qA|j|tjjkrw|j |||j|}|j|qA|jd|jd|j d|j dS)z Loads serialized engine, creates execution context and activates it Args: plan_path: path to serialized TRT engine. logger: optional logger object trt_compilezLoading TensorRT engine: NrzLoaded TensorRT engine: z . Inputs: z Outputs: ) plan_pathrloggerinforrenginertensorscuda_graph_instancecreate_execution_contextcontext input_names output_namesdtypes cur_profile input_tabler$r3num_io_tensorsget_tensor_moder TensorIOModeINPUTr5OUTPUTget_tensor_dtype)selfrIrJ dtype_dictidxbindingdtyper"r"r#__init__s2    zTRTEngine.__init__cCs~|j}t|jD]4\}}t||}||jvs"t|j|j|krwq}t|dksLJdS)z Sets input bindings for TRT engine according to feed_dict Args: feed_dict: a dictionary [str->Tensor] stream: CUDA stream to use csTjD]$}j|d}|dur'|}|j}||||qdSN)rQgetrUrhrfset_input_shaperirj)r_rlrfrk feed_dictr\r"r#try_set_inputss  z,TRTEngine.set_inputs..try_set_inputsTr-rN) rLrPrTrBnum_optimization_profilesset_optimization_profile_async Exception infer_shapesr4)r\rrstreame last_profilers next_profileleftr"rqr# set_inputss(    zTRTEngine.set_inputsFcCs|rO|jdurtt|j|tt||jS|j|}|s&tdtt|tj j |j|tt |}tt |d|_|j d|jS|j|}tt||sbtd|jS)z Runs TRT engine. Args: stream: CUDA stream to run on use_cuda_graph: use CUDA graph. Note: requires all inputs to be the same GPU memory between calls. NzERROR: inference failed.rzCUDA Graph captured!)rNrAcudartcudaGraphLaunchcudaStreamSynchronizerPexecute_async_v3 ValueErrorcudaStreamBeginCapturecudaStreamCaptureMode cudaStreamCaptureModeThreadLocalcudaStreamEndCapturecudaGraphInstantiaterJrKrM)r\rxuse_cuda_graphnoerrorgraphr"r"r#infers.     zTRTEngine.inferrn)F)rCrDrErFrarmr}rr"r"r"r#rG}s   $rGcCst|tjr|St|S)zI Creates a new tensor from d, returns d if d is already a tensor ) isinstancerTensortensorcuda)dr"r"r# make_tensorsrcCspi}|D]1}||}|dur5t|tst|tr/tt|D]}t||||d|<qqt|||<q|S)z; Simulates list/tuple unrolling during ONNX export N_)rrdtupler3r4r)rQ input_exampleunrolled_inputnamevalr<r"r"r# unroll_inputs rretList[torch.Tensor] output_listsList[List[int]]return3Tuple[Union[torch.Tensor, List[torch.Tensor]], ...]c Cst}d}tt|D]}||}t|dkst|dksJt|dks+|ddkr9g|||R}|d}q |ddkrUg|||||dR}||d}q |ddkrt}t|}tt|d|dD]M}||} t| dkst| dksJt| dks| ddkr|d}g|||R}ql| ddkr|| d}g||||| dR}qltdg|||||dddR}|Sq |S)a) Implements parsing of 'output_lists' arg of trt_compile(). Args: ret: plain list of Tensors output_lists: list of output group sizes: to form some Lists/Tuples out of 'ret' List, this will be a list of group dimensions, like [[], [5], [-1]] for returning Tensor, list of 5 items and dynamic list. Format: [[group_n] | [], ...] [] or group_n == 0 : next output from ret is a scalar group_n > 0 : next output from ret is a list of group_n length group_n == -1: next output is a dynamic list. This entry can be at any position in output_lists, but can appear only once. Returns: Tuple of Union[torch.Tensor, List[torch.Tensor]], according to the grouping in output_lists rr-zTwo -1 lists in outputN)rr3r4r) rrgroupscurr<gl rev_groupsrcurrlrglr"r"r# parse_groupss:      $rc@s^eZdZdZ              dddZdd Zd d Zd d ZddZddZ dS) TrtCompilerz This class implements: - TRT lazy persistent export - Running TRT with optional fallback to Torch (for TRT engines with limited profiles) fp16onnxNFcCsddg}||vrtd|d|dgd}||vr&td|d|d||_||_||_|du|_|p7g|_|pTRT). This is the most stable and efficient option. 'torch_trt' may not work for some nets. Also AMP must be turned off for it to work. input_names: Optional list of input names. If None, will be read from the function signature. output_names: Optional list of output names. Note: If not None, patched forward() will return a dictionary. output_lists: Optional list of output group sizes: when forward() returns Lists/Tuples, this will be a list of their dimensions, like [[], [5], [-1]] for Tensor, list of 5 items and dynamic list. export_args: Optional args to pass to export method. See onnx.export() and Torch-TensorRT docs for details. build_args: Optional args to pass to TRT builder. See polygraphy.Config for details. input_profiles: Optional list of profiles for TRT builder and ONNX export. Each profile is a map of the form : {"input id" : [min_shape, opt_shape, max_shape], ...}. dynamic_batchsize: A sequence with three elements to define the input batch size range for the model to be converted. Should be a sequence like [MIN_BATCH, OPT_BATCH, MAX_BATCH]. [note]: If neither input_profiles nor dynamic_batchsize specified, static shapes will be used. use_cuda_graph: Use CUDA Graph for inference. Note: inputs have to be the same GPU memory between calls! timestamp: Optional timestamp to rebuild TRT engine (e.g. if config file changes). fallback: Allow to fall back to Pytorch when TRT inference fails (e.g, shapes exceed max profile). r torch_trtz)trt_compile(): 'method' should be one of z, got: .)fp32tf32rbf16z,trt_compile(): 'precision' should be one of NFrHr-)!rrI precisionmethod return_dictrRrr6r' export_args build_argsrLrfallbackdisabledrrJinspectgetfullargspecforwardargspecargsdefaultsr3r4rrQ old_forwardospathexistsgetmtimeremove)r\modelrIrrrQrRrrrinput_profilesr'r timestamprforward_overriderJ method_valsprecision_valsr<rr"r"r#raJsJ.       ( zTrtCompiler.__init__cCs,i}t|D] \}}|j|}|||<q|Srn)rcrQ)r\r trt_inputsr<inp input_namer"r"r#_inputs_to_dicts   zTrtCompiler._inputs_to_dictc Csz:t|j|j|_i}|jjD]}|dr"||jvr"|dd}n|}|||<q||j_|jd|jjWdStyV}z|jd|WYd}~dSd}~ww)zO Loads TRT plan from disk and activates its execution context. __r.NzEngine loaded, inputs:z$Exception while loading the engine: ) rGrIrJrLrQ startswithrUrKrv)r\rUr orig_nameryr"r"r# _load_engines   zTrtCompiler._load_enginec CsV|j}||t|dkr||||jdur|js|j}|j|_z4||jdurX| }t | ||Wdn1sHwY||jdusXJWn$t y}}z|jrq|jd|d|_n|WYd}~nd}~ww|js|js|D]}~qt j||_zj|jdurtYt j} t jj| d} |jt|j|| j|jj| d| t j|jj| j|jd} |j st!| "} |j#rt$| |j#} n t| dkr| d} | WdWS1swYWn$t y"}z|jr|jd|d n|WYd}~nd}~ww|j|i|S) af Main forward method: Builds TRT engine if not available yet. Tries to run TRT engine If exception thrown and self.callback==True: falls back to original Pytorch Args: Passing through whatever args wrapped module's forward() has Returns: Passing through wrapped module's forward() return value(s) rNzFailed to build engine: T)rb)rr-z Exception: z Falling back to Pytorch ...)%rupdater4rrLrrrrcopyrno_grad_build_and_savervrrJrK parametersr empty_cachelock_smcurrent_deviceStreamr}rrQ cuda_streamrm wait_streamcurrent_streamrrrrdvaluesrr) r\rargvkwargsr new_forwardrryparamrbrxrr"r"r#rsp            " zTrtCompiler.forwardc Csg}|jD]"}t}|D]\}}|j||d|d|ddq||q|j}|jdk|d<|jdkr>d|d<n |jd krGd|d <|j d |d |j t |t j jgd }t|tdd |i|dS)z[ Builds TRT engine from ONNX file at onnx_path and saves to self.plan_path rr-r.)minoptmaxrrrTrzBuilding TensorRT engine for z: )flagsr6)configNr")r6ritemsaddr5rrrrJrKrIrrOnnxParserFlagNATIVE_INSTANCENORMrr) r\ onnx_pathr6r8pidrrnetworkr"r"r# _onnx_to_trts       zTrtCompiler._onnx_to_trtc sjdurdSj}d}t|t|jdkrVtjg}jdkr)|tj n jdkr4|tj t | }ddfdd|D}t j|d f||d |}njrtjd krftd td krptdi|D]6\}} fdd} t| t st| trtt| D]} | |d| | | qqvt| tjr| || qvg_tj_tjd kr|djitp} |ddrd} n t tj| } t!t"| d}j#$d|ddj%djd|tj&j'||f|f| j%d|t(r(d dl)m*}m+}m,}|||dd}|||j#$d-|}Wdn 1s>wY|rQt.j/d 0|dSdS)!z If TRT engine is not ready, exports model to ONNX, builds TRT engine and saves serialized TRT engine to the disk. Args: input_example: passed to onnx.export() NrrrcSs t||\}}}tj|||dS)N) min_shape opt_shape max_shape)r2rInput)r%r'r/r0r1r"r"r#get_torch_trt_input2sz8TrtCompiler._build_and_save..get_torch_trt_inputcsg|] }|jjqSr")rfr').0r<)rr\r"r# 8sz/TrtCompiler._build_and_save..r) arg_inputsenabled_precisionsrzEERROR: Both dynamic_batchsize and input_profiles set for TrtCompiler!z&dynamic_batchsize has to have len ==3 csR|j}t|dkr'|dd}dg|dg|dg|g|<dSdS)Nrr-r.)rfr4)rrsh)dbsr8r"r# add_profileJs   0z0TrtCompiler._build_and_save..add_profilerr7dynamoFz model.onnxz Exporting to z: z output_names=z input_names=z export args: )rQrR)fold_constantsonnx_from_path save_onnxi$)size_thresholdzExport to ONNX successful.wb)1rLrr rrrrrr5rrrdrrconvert_method_to_trt_enginer'r4r6rrrrr3rr=r7rtempfileTemporaryDirectoryrorrQkeysstrrrJrKrRrexportpolygraphy_importedpolygraphy.backend.onnx.loaderrrrropenrIwrite)r\rrr engine_bytesrinputs tt_inputsrrrr<tmpdirrQrrrr onnx_modelr")rrr8r\r#rs                zTrtCompiler._build_and_save)rrNNNNNNNFNFNN) rCrDrErFrarrrrrr"r"r"r#rBs,  XF rcOs|j|||S)zk Patch function to replace original model's forward() with. Redirects to TrtCompiler.forward() ) _trt_compilerr)r\rrr"r"r# trt_forwardzsrrtorch.nn.Module base_pathrrDict[str, Any] | None submoduleUnion[str, List[str]] | NonerJ Any | Nonec sdddddd}|pi|trttrttjrttj|r:t tj |}dvr6t t d|}|d<fdd }fd d |d urmt |t rS|g}|D]}||\} } |t| | |d |qU|S||||Spytdd|S)aJ Instruments model or submodule(s) with TrtCompiler and replaces its forward() with TRT hook. Note: TRT 10.3 is recommended for best performance. Some nets may even fail to work with TRT 8.x Args: model: module to patch with TrtCompiler object. base_path: TRT plan(s) saved to f"{base_path}[.{submodule}].plan" path. dirname(base_path) must exist, base_path does not have to. If base_path does point to existing file (e.g. associated checkpoint), that file becomes a dependency - its mtime is added to args["timestamp"]. args: Optional dict : unpacked and passed to TrtCompiler() - see TrtCompiler above for details. submodule: Optional hierarchical id(s) of submodule to patch, e.g. ['image_decoder.decoder'] If None, TrtCompiler patch is applied to the whole model. Otherwise, submodule (or list of) is being patched. logger: Optional logger for diagnostics. Returns: Always returns same model passed in as argument. This is for ease of use in configs. rrobey)builder_optimization_levelprecision_constraints)rrrrcsFt|ds!|j|_t||dfdi}||_tt||_dSdS)Nrz.planrJ)hasattrr orig_forwardrrrr)rrwrapper)rrJr"r#wraps ztrt_compile..wrapcsJ|d}|dkr!|d|}t||}||dd}||S||fS)Nrrr-)findgetattr)parentrr^ parent_name)find_subr"r#r&s    ztrt_compile..find_subNrrHzSTensorRT and/or polygraphy packages are not available! trt_compile() has no effect.)r trt_importedr rr is_availablerrrr*rrrrr#rwarning) rrrrrJ default_argsrr!sr$subr")rr&rJr#rHs4     rH)r%r&r'r()rrrrrr)NNN) rrrrrrrrrJrrr)9 __future__rrrr threading collectionsrloggingrpathlibrtypesrtypingrrr r r r rnemo.utils.export_utilsr rnemo.utils.import_utilsrrr polygraphy.backend.commonrpolygraphy.backend.trtrrrrrrr'rrr~Lockrr$r2r=rArvrBrGrrrrrrHr"r"r"r#sL             | 2: