o ٷi/@sddlZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z ddlmZmZddlmZmZdd lmZeeZgd Zgd ZGd d d ZdS)N)Path)float_to_float16_max_diff) OnnxModel)optimize_model) T5DecoderT5DecoderHelper)T5EncoderDecoderInitT5EncoderDecoderInitHelper)MT5ForConditionalGenerationT5ForConditionalGeneration)InferenceSession)zt5-smallzt5-basezt5-largezt5-3bzt5-11b)zgoogle/mt5-smallzgoogle/mt5-basezgoogle/mt5-largez google/mt5-xlzgoogle/mt5-xxlc@s<eZdZe  d.dededededef dd Ze  d/ded ed ejd ededede ee e BffddZ e    d0de e Bd ejdededededefddZ e   d1dedeedBdedefd d!Ze    d2ded"ed#ed$ed%eded&ed'ed(efd)d*Zede e Bd+ed ejdefd,d-ZdS)3T5HelperF output_dirmodel_name_or_pathsuffix new_folderreturncCs^|}tj|rt|jd}n|dd||7}|r$tj||n|}tj||dS)aBuild onnx path Args: output_dir (str): output directory model_name_or_path (str): pretrained model name, or path to the model checkpoint suffix (str, optional): suffix like "_encoder" or "_decoder_fp16" will be appended to file name. Defaults to None. new_folder (bool, optional): create a new directory for the model. Defaults to False. Returns: str: path of onnx model /z.onnx)ospathisdirrpartssplitjoin)rrrr model_name directoryr`/home/ubuntu/.local/lib/python3.10/site-packages/onnxruntime/transformers/models/t5/t5_helper.py get_onnx_path!s zT5Helper.get_onnx_patht5 cache_dirdevice model_typestate_dict_pathencoder_decoder_initc Cs|dkr tj||d}n|dkrtj||d}ntd|r&|t|t|j|j |j }| |t |j|j|j |j d| d}|rIdnd} | |d |iS) a{Load model given a pretrained name or path, then build models for ONNX conversion. Args: model_name_or_path (str): pretrained model name or path cache_dir (str): cache directory device (torch.device): device to run the model model_type (str, optional): model type "t5" or "mt5" state_dict_path(str, optional): state dictionary path encoder_decoder_init (bool, optional): combine encoder and decoder kv cache initialization into one model. Returns: Dict[str, torch.nn.Module]: mapping from name to modules for ONNX conversion. r!)r"mt5z only support mode_type=t5 or mt5N)decoder_start_token_idoutput_cross_onlyr&encoderdecoder)r from_pretrainedr ValueErrorload_state_dicttorchloadrr+lm_headconfigevaltorr*) rr"r#r$r%r&modelr+r* encoder_namerrr load_model>s& zT5Helper.load_modelTr5onnx_model_pathverboseuse_external_data_formatuse_decoder_input_idsuse_int32_inputsc Cs<t|trt|||||||dSt||||||dS)N) isinstancerr export_onnxr)r5r#r8r9r:r;r<rrrr>ls$  zT5Helper.export_onnxN onnx_model op_block_listforce_fp16_logitsuse_symbolic_shape_infercspdurgddd|D}t}||}td|d||jdj}d}|} || vs9J| |} d} | j d krw| } td | jd} | j D] } | | } | duranqTt | }t d | jd ||d k}n td| j d| jg}g|s| dur|s|g}| jgdvr|}d}d}|d}|rd}|D]=} | jvr|| | }|| |}fdd|D}fdd|D}t|t|dkr| j|d7}d}q|st||}td|d|td||d}td||r%|jdddi||Sddlm}||jfddi||S)aConvert model to mixed precision. It detects whether original model has fp16 precision weights, and set parameters for float16 conversion automatically. Args: onnx_model (OnnxModel): optimized ONNX model op_block_list (List[str], optional): operators need to run in fp32. force_fp16_logits (bool, optional): force logits and last MatMul node to be in float16. Defaults to False. use_symbolic_shape_infer (bool, optional): use symbolic shape inference to convert float to float16. Defaults to True. Returns: parameters(dict): a dictionary of parameters used in float16 conversion N)SimplifiedLayerNormalization SkipSimplifiedLayerNormalizationReluAddcSsh|]}|jqSrop_type).0noderrr sz0T5Helper.auto_mixed_precision..z fp32 op: z fp16 op: rFMatMulz#Found last MatMul node for logits: z3max diff of converting weights in last MatMul node z: gư>z-Failed to find MatMul node for logits. Found z of node rFTc"g|] }|jvs |vr|qSrrG)rIchildnode_block_listr@rr z1T5Helper.auto_mixed_precision..crMrrG)rIparentrOrrrQrRz#node counter of Add operator: fp32=z fp16=znode_block_list: ) keep_io_typesr@rPforce_fp16_initializersz!auto_mixed_precision parameters: rB)convert_float_to_float16disable_shape_inferr)nodesset differenceloggerinfographoutputnameoutput_name_to_noderHinputget_initializerrdebugwarninginput_name_to_nodesget_nodes_by_op_type get_parents get_childrenlenappendrWfloat16r5)r?r@rArB op_full_set fp32_op_set fp16_op_setlogits_output_nameis_weight_fp16_precisionrarJlast_matmul_node initializerrbmax_diffrUrffp32_addchanged add_nodesparentschildrenblocked_childrenblocked_parentsfp16_add parametersrWrrOrauto_mixed_precisions               zT5Helper.auto_mixed_precisionoptimized_model_path is_float16num_attention_heads hidden_sizer~use_gpu force_fp16_ioc Csrddlm} d} |r| d} | | _t|d||d| |d} |r/|r)tj| |dn| j|d| j||dd dS) zHOptimize ONNX model with an option to convert it to use mixed precision.r) FusionOptionsNr!)r$ num_headsr opt_leveloptimization_optionsr)rA)cast_input_outputT)all_tensors_to_one_file)fusion_optionsrenable_skip_layer_normrr r~ convert_model_float32_to_float16save_model_to_file) r8rrrrr:r~rrrrmrrr optimize_onnxs&   zT5Helper.optimize_onnx ort_sessioncCs*t|tr t||||St||||S)zQCompare the result from PyTorch and OnnxRuntime to verify the ONNX model is good.)r=rr verify_onnxr)r5rr#r<rrrr#s zT5Helper.verify_onnx)rF)r!rF)TFTF)NFT)FTFF)__name__ __module__ __qualname__ staticmethodstrboolr r/r#dictrrr7r>rlistr~intrr rrrrrr s  -   p 'r )loggingrpathlibrr/rlrr?r optimizerr t5_decoderrrt5_encoder_decoder_initrr transformersr r onnxruntimer getLoggerrr\PRETRAINED_T5_MODELSPRETRAINED_MT5_MODELSr rrrrs