o }oi@sddlmZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZdd gZejjejjd ZeGd d d ZGd d d e ZGddde ZdS)) dataclass)DictOptional)nn)MultiLayerPerceptron) Classifier) typecheck) LogitsType LogprobsType NeuralTypeBertPretrainingTokenClassifierTokenClassifier)gelureluc@sJeZdZUdZeed<dZeed<dZe ed<dZ e ed<dZ e ed <d S) TokenClassifierConfig num_layersr activationT log_softmaxdropoutuse_transformer_initN) __name__ __module__ __qualname__rint__annotations__rstrrboolrfloatrr r h/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/token_classifier.pyrs    rczeZdZdZedeeeeffddZ     dd e d e d e d ed e de de ddffdd Z eddZZS)r z` A module to perform token level classification tasks such as Named entity recognition. returncC&|js dtdtiSdtdtiSz= Returns definitions of module output ports. logits)BTC log_probsrr r r selfr r r! output_types,zTokenClassifier.output_typesrrTr hidden_size num_classesrrrrrNcs:tj||d||_t|||||d|_|j|ddS)q Initializes the Token Classifier module. Args: hidden_size: the size of the hidden dimension num_classes: number of classes num_layers: number of fully connected layers in the multilayer perceptron (MLP) activation: activation to usee between fully connected layers in the MLP log_softmax: whether to apply softmax to the output of the MLP dropout: dropout to apply to the input hidden states use_transformer_init: whether to initialize the weights of the classifier head with the same approach used in Transformer r0rrrrrN)super__init__rrmlp post_initr-r0r1rrrrr __class__r r!r76s  zTokenClassifier.__init__cCs||}||}|Sa3 Performs the forward step of the module. Args: hidden_states: batch of hidden states (for example, from the BERT encoder module) [BATCH_SIZE x SEQ_LENGTH x HIDDEN_SIZE] Returns: logits value for each class [BATCH_SIZE x SEQ_LENGTH x NUM_CLASSES] )rr8)r- hidden_statesr&r r r!forwardTs zTokenClassifier.forwardrrTrTrrr__doc__propertyrrrr r.rrrr7rr? __classcell__r r r;r!r 's8  cr")r zT A module to perform token level classification tasks for Bert pretraining. r#cCr$r%r+r,r r r!r.gr/z+BertPretrainingTokenClassifier.output_typesrrTrr0r1rrrrrNcsztj||d||_|tvrtd|dt|||_t||_tj |dd|_ t |||||d|_ |j |ddS) r2r3z activation "z " not foundg-q=)epsr4r5N)r6r7rACT2FN ValueErrorrLineardenseact LayerNormnormrr8r9r:r;r r!r7qs  z'BertPretrainingTokenClassifier.__init__cCs6||}||}||}||}||}|Sr=)rrIrJrLr8)r-r> transformr&r r r!r?s    z&BertPretrainingTokenClassifier.forwardr@rAr r r;r!r bs8  %N) dataclassesrtypingrrtorchrnemo.collections.common.partsr.nemo.collections.nlp.modules.common.classifierrnemo.core.classesrnemo.core.neural_typesr r r __all__ functionalrrrFrr r r r r r!s     ;