o Ơik@sddlmZddlZddlZddlZddlmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZdd lm Z m!Z!m"Z"e#e$Z%e j&d d e j&dd d"ddZ'e j&dd d"ddZ(edfd#d d!Z)dS)$) annotationsN)AnyCallableSequence) aot_autograd)detect_fake_mode)aot_export_joint_simple)is_tegra_platform)CompilationSettings)compile_module)get_decompositions post_lowering remove_detachremove_sym_nodesrepair_input_aliasing)parse_dynamo_kwargsprepare_inputs set_log_leveltensorrt)nametorch_tensorrtgmtorch.fx.GraphModule sample_inputs Sequence[Any]kwargsrreturntorch.nn.ModulecKsTd|vrd|dvr|ddsd|vr|drttjtjt}|||fi|S)Noptionsdebug)rloggerparentloggingDEBUGaot_torch_tensorrt_aten_backend)rrrDEFAULT_BACKENDr&Z/home/ubuntu/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/backend/backends.pytorch_tensorrt_backend s  r(aot_torch_tensorrt_atenc  st|\}}|jr=tdtjt||d}i}t|j|d<dd|dD}|D]}|d|=q*t ||dd||St sVddl m t fd d |DrVtd |jr^td t||||S) Nz@Wrapping the backend with aot_autograd for Distributed examples settings engine_cachedecompositionscSsh|] }d|jvr|qS)detach)_name).0keyr&r&r' Gs  z2aot_torch_tensorrt_aten_backend..) fw_compilerr-rDTensorc3s|]}t|VqdS)N) isinstance)r0tensorr4r&r' Wsz2aot_torch_tensorrt_aten_backend..zIt is recommended to run the model with use_distributed_mode_trace = True since there are distributed tensors in the input which is not supported in aot_export_joint_simplez~The offload_module_to_cpu option is set, but it is being ignored since the torch_compile backend does not support this feature)ruse_distributed_mode_tracer r functoolspartial_pretraced_backendr "enable_experimental_decompositionsrr torch.distributed.tensorr5anywarningoffload_module_to_cpu) rrrr+r,_pretraced_backend_autogradsettings_aot_autograd to_deleter1r&r4r'r$2sD   r$r+r r,)torch.fx.GraphModule | Callable[..., Any]c Csztdt|jt|}tjj|dd|qt ||t |||dd|D}t |||j sAt ||dt|jd}tdt|jt||}td t|jt|dd }|jrhtd |jrstd d|_t||||d }|WdWdWS1swYWdWdS1swYWdSttfy|jstjddd|YStdw)a'Helper function to manage translation of traced FX module to TRT engines Args: module: FX GraphModule to convert inputs: Inputs to the module settings: Compilation settings engine_cache: Engine cache instance Returns: Compiled FX GraphModule zPre-AOT Autograd graph: allow_non_fake_inputsTcSsg|] }t|tjr|qSr&)r6torchTensor)r0inputr&r&r' s  z&_pretraced_backend..F) trace_jointr-zPost-AOT Autograd graph: zLowered Input graph: )disable_memory_format_checkz]require_full_compilation arg is not applicable for torch.compile with backend='torch_tensorrtzfstrip_engine_weights=True is not supported for torch.compile(). It will be set to False automatically.r*Nz^TRT conversion failed on the subgraph. See trace above. Returning GraphModule forward instead.)exc_infozHalting compilation on build failure since pass_through_build_failures was specified as True. To return the default Torch implementation and avoid halting compilation on engine build failures, specify pass_through_build_failures=False.)r rstrgraphrunittestmockpatchobjectrrrr9rr r=r rrequire_full_compilationr@strip_engine_weightsr AssertionError RuntimeErrorpass_through_build_failurescritical)rrr+r, fake_mode torch_inputstorchtrt_inputs trt_compiledr&r&r'r<cst    X4r<)rrrrrrrr) rrrrr+r r,rrrE)* __future__rr:r"rPtypingrrrrG torch._dynamo_dynamotdtorch._dynamo.backends.commonrtorch._dynamo.utilsrtorch._functorch.aot_autogradrtorch_tensorrt._utilsr torch_tensorrt.dynamor torch_tensorrt.dynamo._compilerr torch_tensorrt.dynamo.loweringr r rrrtorch_tensorrt.dynamo.utilsrrr getLogger__name__r register_backendr(r$r<r&r&r&r's0             3