o }oi @sgdZddlZddlZddlmZmZmZmZmZddl Z ddl Z ddl m mZddlmZddlmZddlmZddlmZmZddl mZdd lmZmZdd lmZdd lm Z dd l!m"Z"d4d ee#e$fdee%de j&fddZ'dee#de$fddZ(dedeee#e)ffddZ*   d5dee#dee#de$dee$e#fde%de$f d d!Z+dee#dee#de$fd"d#Z,d6d$d%Z-d&d'Z.d(d)Z/d7d*e$d+e%d,e%defd-d.Z0d/ed0ed1ee$e jffd2d3Z1dS)8)torch_dtype_from_precisionlist2str tensor2listplot_confusion_matrixget_classification_reportload_state_dict_helperN)CallableDictListOptionalUnion)Trainer)pyplot) DictConfig)classification_reportconfusion_matrix)Tensor)ApproxGELUActivationerf_gelu) openai_gelu) squared_relu)logging precisionmegatron_amp_O2returncCsP|dur |dur tjS|dvrtjS|dvrtjS|dvr tjStd|d)zMMapping from PTL precision types to corresponding PyTorch parameter datatype.NF)bf16z bf16-mixed)16z16-mixed) 32z32-truez"Could not parse the precision of `z` to a valid torch.dtype)torchfloat32bfloat16float16 ValueError)rrr%Z/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/parts/utils_funcs.pyr+srlcCsddd|DS)zConverts list to a string cSsg|]}t|qSr%str).0xr%r%r& <zlist2str..)join)r'r%r%r&r:srtensorcCs|S)zConverts tensor to a list)detachcputolist)r0r%r%r&r?srFlabelspreds graph_fold label_ids normalizeprefixcsdur t|||dSt|t|BfddDfddDfddtD}d}t||}|r`|jdd ddtjf} t | d kd| } | d | }d |}t } | d } | |} | tdt|| tdt|| jdg|dd| dg|| d| dtj|dd| | ||}tj||dtd} t | t !d| dS)aQ Plot confusion matrix. Args: labels: ground truth labels preds: model predictions graph_fold: path to a folder to store confusion matrix plot label_ids: str label to id map, for example: {'O': 0, 'LOC': 1} normalize: whether to normalize confusion matrix prefix: prefix for the plot name Ncs"i|] \}}|vr||qSr%r%r+kv)r8 used_labelsr%r& \s"z)plot_confusion_matrix..csi|]}||qSr%r%)r+r<)r8r%r&r?^scsg|]}|qSr%r%)r+id) ids_to_labelsr%r&r-_r.z)plot_confusion_matrix..Confusion_matrix)axisrfloat Normalized_or4Z)rotationTrue PredictedTexist_ok_ %Y%m%d-%H%M%SzConfusion matrix saved to )"_plot_confusion_matrixsetitemssortedvaluesrsumnpnewaxiswhereastypepltfigure add_subplotmatshow set_xticksarangelen set_yticksset_xticklabelsset_yticklabels set_ylabel set_xlabelosmakedirscolorbarstrippathr/timestrftimesavefigrinfo)r5r6r7r8r9r:classestitlecmsumsfigaxcaxfig_namer%)rAr8r>r&rDs8        rcCst||}td|t}|d}||}td||t dt dt j |ddt t j|tdd S) z Plot confusion matrix Args: labels: ground truth labels preds: model predictions graph_fold: path to a folder to store confusion matrix plot zConfusion matrix: rGz"Confusion matrix of the classifierrLrKTrMrPN)rrror[r\r]r^rqrixlabelylabelrgrhrnrkr/rlrm)r5r6r7rrrtrurvr%r%r&rQ~s       rQcsDt|t|Bfddt|dddD}t|||d|dS)z' Returns classification report cs,g|]\}}|vr|dt|dqS)z (label id: )r)r;r>r%r&r-s z-get_classification_report..cSs|dSNrCr%)itemr%r%r&sz+get_classification_report..)key) target_namesdigits output_dict)rRrTrSr)r5r6r8r labels_namesr%r{r&rs  rcCstjtjdkSr|)r distributedget_rankget_world_sizer%r%r%r& is_last_ranksrcCstjdSr|)r rrr%r%r%r& get_last_ranksr activationr onnx_safecCsgd}||vrtd|d|d|rt}|S|dvr"tj}|S|r(t}|S|dvr1tj}|S|dvr:tj}|S|dkrBt}|S|d krHt}|S) aConverts an activation function represented as a string to a function. Args: activation (str): string representation of an activation function, typically gotten from the model config. openai_gelu (bool): whether to use the OpenAI GELU implementation. Used with HF compatibility. onnx_safe (bool): whether to use the ONNX-compatible implementation of GELU. Returns: Callable: the activation function. ) gelugegluregluswiglu squared-relu fast-geglu fast-swiglu fast-reglu approx-geluzUnsupported activation z. Supported activations: r()rrr)rr)rrrr) r$openai_gelu_funcFrrrelusilurr)rrrsupported_activationsactivation_funcr%r%r&activation_to_funcs0     rcfgtrainer state_dictcCsj|||}|j|dd\}}|r%dd|D}|r%tdtd||r3tdtd||S) zILoad state_dict for converted community, for example, HuggingFace models.F)strictcSsg|] }|ds|qS) _extra_state)endswith)r+rr%r%r&r-sz*load_state_dict_helper..zOMissing keys were detected during the load, something has gone wrong. Aborting.zMissing keys: z@Unexpected keys were detected which should not happen. Aborting.zUnexpected keys: )load_state_dictrcritical RuntimeError)clsrrrmodel missing_keysunexpected_keysmissing_keys_non_extrar%r%r&rs   r)N)NFr4)F)FF)2__all__rgrltypingrr r r r numpyrWr torch.nn.functionalnn functionalr!lightning.pytorch.trainer.trainerr matplotlibrr[omegaconf.dictconfigrsklearn.metricsrrr2nemo.collections.nlp.modules.common.megatron.utilsrrrrr nemo.utilsrintr*booldtyperrrErrrQrrrrrr%r%r%r&sR        &   : $0