o }oiG @snddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddlm mZddlmZmZmZzddlZdaWn eefyRdaYnwGdd d eZejejejd ZGd d d e jZGd dde jZde fddZ!de de fddZ"ddZ#ddZ$ddZ%ddZ&dPdd Z'dPd!d"Z(dPd#d$Z)dPd%d&Z*dZ+zddd'l,m-Z-dd(l.m/Z/dd)l0m1Z1m2Z2dd*l3m1Z4dd+l5m6Z6dd,l7m8Z8m9Z9d-e jd.ee j:fd/d0Z;d-e jfd1d2Zd-e jd.ee j=fd5d6Z?e;e;e;e;e>e>e?ed?ZEd@e jdAee e jffdBdCZF dQd@e jdDee ee jgee jffd.e jfdEdFZGdGe jfdHdIZHiZId@e jd.e jfdJdKZJd@e jfdLdMZKdNdOZLdS)RN) nullcontext)Enum)CallableDictOptionalType) CastToFloatCastToFloatAllloggingTFc@seZdZdZdZdZdS) ExportFormatzAWhich format to use when exporting a Neural Module for deploymentN)__name__ __module__ __qualname____doc__ONNX TORCHSCRIPTrrK/home/ubuntu/.local/lib/python3.10/site-packages/nemo/utils/export_utils.pyr #sr )z.ptz.tsz.onnxcs&eZdZdfdd ZddZZS) TorchRMSNormư>cst||_||_dS)z( LayerNorm without bias N)super__init__weightvariance_epsilon)selfreps __class__rrr2s  zTorchRMSNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)Nr T)keepdim) totorchfloat32powmeanrsqrtrrdtypefloat16bfloat16)r hidden_statesvariancerrrforward:s  zTorchRMSNorm.forward)rrrrrr- __classcell__rrrrr1srcs&eZdZfddZdddZZS)LinearWithBiasSkipcs$tt|||_||_||_dSN)rr0rbiasr skip_bias_add)rrr2r3rrrrEs zLinearWithBiasSkip.__init__NcCs:|dur|j}|jrt|||jfSt|||jdfSr1)rr3Flinearr2)rxrrrrr-Ks zLinearWithBiasSkip.forwardr1r.rrrrr0Ds r0filenamecCs>tj|\}}zt|WStytd|dw)Nz Export file z4 extension does not correspond to any export format!)ospathsplitext _EXT_DICTlowerKeyError ValueError)r7_extrrrget_export_formatSs  rAoutputprependcCs8|dkr|Stj|\}}|d|}tj||S)Nr-)r8r9splitjoin)rBrCr9r7rrraugment_filename[s rGcCst|dr|jS|jSNforward_for_export)hasattrrIr-)rrrrforward_methodds rKcCs<t|}d}t|dr|j}|j}||_||fSd}||fSrH)typerJrIr-)rtpold_forward_methodrKrrrwrap_forward_methodks rOcCs6t|}i}t|dtr|d}|dd}||fS)Nr )list isinstancedict) input_example input_list input_dictrrrparse_input_examplews  rVcCsi}|s||t||D] \}}|||<q|St|D]+}d}||vr4||}nt|dkrB|}||vrN|durN|||<q#|S)Nr)extendvalueszipcpunumpyreversedlenpop)ort_input_names input_namesrUrTodictkvvalrrrto_onnxrt_inputs   re{Gz?c Csd}|D]8}t|\}}tjjddd |j|i|}tj|} |o,t| ||||}Wdn1s7wYq|rAdnd} t d|d| |S) NTcudaF)enabledSUCCESSFAILzTorchscript generated at z% verified with torchscript forward : ) rVr#ampautocastr-jitloadrun_ts_and_comparer info) modelrBinput_examplescheck_toleranceall_goodrSrTrUoutput_examplets_modelstatusrrrverify_torchscripts    rxcCst|}dd|jjD}ts#td|dtjj|dddSt }t j j |_ t j||dgd}~d} |D](} t| \} } |j| i| } t| tsT| f} t||| | }| oct||| |} q<| rid nd }td|d || S) NcSsg|]}|jqSr)name).0noderrr sz"verify_runtime..zONNX generated at z9, not verified - please install onnxruntime_gpu package. T) full_checkCUDAExecutionProvider) sess_options providersrirjz verified with onnxruntime : )onnxrngraphinput ort_availabler warningchecker check_model onnxruntimeSessionOptionsGraphOptimizationLevelORT_ENABLE_BASICgraph_optimization_levelInferenceSessionSerializeToStringrVr-rQtuplererun_ort_and_comparerp)rqrBrrr`rs onnx_modelr_onnx_session_optsessrtrSrTrUru ort_inputrwrrrverify_runtimes.      rc Cs||i|}d}t|D]K\}}||} t| rX|d} td|d| jdd} ztj| | ||ds|j j |j d}}}ndS| }tj||||j|jd}|j|dd|S)z Replaces Apex's FusedLayerNorm with nn.LayerNorm. This is required for ONNX export. Args: n: the FusedLayerNorm pytorch module to replace Returns: Equivalent LayerNorm module TN)relementwise_affinedevicer(strict)next parametersrQrrnormalized_shaperrMCoreFusedLayerNormrrrepsilon state_dictnn LayerNormrr(load_state_dict)rprraffinen_statemodrrrreplace_FusedLayerNorms   rcCs:t|}t|trt|d|j|j}|SdS)z Replaces Apex's MixedFusedRMSNorm with equivalent Pytorch layer. This is required for ONNX export. Args: n: the MixedFusedRMSNorm pytorch module to replace Returns: Equivalent module rN) rrrQrrrrr"r)rrrrrrreplace_MixedFusedRMSNorms rcCs\t|tst|tstdt|j}t|j|j |j  |}| }|j |dd|S)z Replaces Apex's ColumnParallelLinear or RowParallelLinear with nn.Linear Args: n: the nn.Module pytorch module to replace Returns: Equivalent Linear module zSThis function can only change the ColumnParallelLinear or RowParallelLinear module.Fr)rQrrr>rrrr0rr2r3r"rr)rdevrrrrrreplace_ParallelLinear%srcCsDt|tstd|j|St|j|j|jd|j|j |j }|S)z Replaces Apex's FusedScaleMaskSoftmax with nn.LayerNorm. This is required for ONNX export. Args: n: the FusedScaleMaskSoftmax module to replace Returns: Equivalent LayerNorm module zEThis function can only change the FusedScaleMaskSoftmax module, got: F) rQrr rr input_in_fp16 input_in_bf16attn_mask_type mask_funcsoftmax_in_fp32scale)rrrrrreplace_FusedScaleMaskSoftmax7s r)rrrrrrrrBaseTDestTcs$dtjdttjffdd }|S)a+ Generic function generator to replace BaseT module with DestT. BaseT and DestT should have same atrributes. No weights are copied. Args: BaseT : module type to replace DestT : destination module type Returns: swap function to replace BaseT module with DestT rrcs.tsdSfddjD}|}|S)Ncsg|]}t|dqSr1)getattr)rzryrrrr|gsz8simple_replace..expansion_fn..)rQ __constants__)rargsrrrrr expansion_fnds z$simple_replace..expansion_fnrModulerrrrrrrsimple_replaceZs rcCs0ddlm}||j|j|jd|j|j|j}|S)z Replaces MatchedScaleMaskSoftmax with exportable softmax layer Args: n: module to replace Returns: exportable module r)MatchedScaleMaskSoftmaxF):nemo.collections.nlp.modules.common.megatron.fused_softmaxrrrrrrr)rrrrrrreplace_MatchedScaleMaskSoftmaxns rcs"dtjdttjffdd }|S)z Generic function generator to replace BaseT module with DestT wrapper. Args: BaseT : module type to replace DestT : destination module type Returns: swap function to replace BaseT module with DestT rrcs |}|Sr1r)rrrrrrsz!wrap_module..expansion_fnrrrrr wrap_modules rrqmappingcCsN|D] \}}|d}|}|ddD]}|j|}q||j|d<q|S)a This function swaps nested modules as specified by "dot paths" in mod with a desired replacement. This allows for swapping nested modules through arbitrary levels if children NOTE: This occurs in place, if you want to preserve model then make sure to copy it first. .Nr )itemsrE_modules)rqrr9new_mod expanded_path parent_modsub_pathrrr swap_moduless  r expansionscCspi}|D]\}}t|j}||vr|||}|r|||<qt|dkr1tdt|dt|||S)a Top-level function to replace modules in model, specified by class name with a desired replacement. NOTE: This occurs in place, if you want to preserve model then make sure to copy it first. Args: model : top level module expansions : replacement dictionary: module class name -> replacement function generator Returns: model, possibly modified in-place rzSwapped z modules) named_modulesrLrr]r rpr)rqrrrymm_typeswappedrrrreplace_moduless     rrcCs tj|Sr1)r#rmscript)rrrr script_modules rcCs0dtdti}t|tt||t|tdS)a, Top-level function to replace 'default set' of modules in model, called from _prepare_for_export. NOTE: This occurs in place, if you want to preserve model then make sure to copy it first. Args: model : top level module Returns: model, possibly modified in-place rN)rrrdefault_Apex_replacementsscript_replacements)rqdefault_replacementsrrrreplace_for_exports   rcCsPddlm}ttjtttjtttjtttjtt|t d}t ||dS)a Function to put additional to/from float32 casts around operations known to require full precision. It was used with an extra post-parse script to have TRT preserve extra precision when --fp16 needed. Should not be needed with TRT 8.6.1 or later. r)MaskedInstanceNorm1d) BatchNorm1d BatchNorm2drInstanceNorm1drN) 'nemo.collections.tts.modules.submodulesrrrrrrrrr r)rqrdefault_cast_replacementsrrradd_casts_around_normss     rc Cs$t|}i}t|jj|D] \}}|||j<qt|jj|D] \}}|||j<q|jjD]8}tt |jD]}|j||vrI||j||j|<q6tt |jD]}|j||vrd||j||j|<qQq-tt |D] } || |jj| _qltt |D] } || |jj| _q~t ||dSr1) rrnrYrrryrBr{ranger]save) rBr` output_namesr rename_mapinpryrrrrrrrename_onnx_ios*    r)rfr1)Mr8 contextlibrenumrtypingrrrrrr#torch.nnrtorch.nn.functional functionalr4 nemo.utilsrr r rr ImportErrorModuleNotFoundErrorr rrr;rrr0strrArGrKrOrVrerxrrorapex_available"apex.contrib.layer_norm.layer_normrapex.normalizationr#apex.normalization.fused_layer_normrr&megatron.core.fusions.fused_layer_normr#megatron.core.fusions.fused_softmaxr$megatron.core.tensor_parallel.layersrrrrrLinearrrrrerrrrrrrrrrrrrrs               44