o ߥib @svddlmZmZmZmZddlZddlmmZ ddlmZddl m Z ddl m Z ddlmZddlmZddlmZdd lmZmZmZmZdd lmZejejejd ejejejd ejejejd ejej ejd Gd d d eZ!ejej"ej"d ejejej"d ejejej"d ejejej"d ejej ej"d GdddeZ#Gdddej$Z%dS))AnyDictListOptionalN)nn)CrossEntropyLoss)ACT2FN)Heads) TorchHead)HEADS)'AttentionTokenClassificationModelOutputModelOutputBase OutputKeysTokenClassificationModelOutput)Tasks) module_namecs>eZdZd fdd Z    d defddZdd ZZS) LSTMCRFHeaddNc s@tj||d|dusJt|d||_t|dd|_dS)N hidden_size num_labelsT batch_first)super__init__rLinearffnCRFcrfselfrrkwargs __class__X/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/heads/crf_head.pyr*s zLSTMCRFHead.__init__inputscKs||j}td|dS)N)losslogits)rlast_hidden_stater)r!r'attention_masklabel label_maskoffset_mappingr"r)r%r%r&forward0s zLSTMCRFHead.forwardcCX|d}tj|jd|jddddf|dddfk}|j||d}|SN)devicer sumlongtorcharangeshaper4rdecodesqueezer!r)r-seq_lensmaskpredictsr%r%r&r;> zLSTMCRFHead.decode)rNNNNN__name__ __module__ __qualname__rr r/r; __classcell__r%r%r#r&r#s rcs<eZdZfddZ    d defddZddZZS) TransformersCRFHeadc s8tjd||d|t|||_t|dd|_dS)NrTrr%)rrrrlinearrrr r#r%r&rRszTransformersCRFHead.__init__Nr'c Ks||j}|durA|}|d} t|} t|jdD]} || ||  d | | d| | d| | ddf<q| }t d||j |j dS)Nr2r)r(r) hidden_states attentions)rIr*r6r7r8 zeros_likeranger: masked_select unsqueezeviewr rJrK) r!r'r+r,r-r.r"r)r?masked_lengths masked_logitsir%r%r&r/Xs(    zTransformersCRFHead.forwardcCr0r1r5r=r%r%r&r;qrAzTransformersCRFHead.decoderBrCr%r%r#r&rHGs  rHcseZdZdZd"dededdffdd Zd#d d Zdefd d Z  d$de j de j de e jdede j f ddZ   d%de j de e jde ede edeeeef ddZ  d&de j de e j de e jddfddZde j de j de jde j fddZde j de jde j fddZ d'de jde jde edeeefddZ d'de jde jdede edeeeef d d!ZZS)(ruConditional random field. This module implements a conditional random field [LMP01]_. The forward computation of this class computes the log likelihood of the given sequence of tags and emission score tensor. This class also has `~CRF.decode` method which finds the best tag sequence given an emission score tensor using `Viterbi algorithm`_. Args: num_tags: Number of tags. batch_first: Whether the first dimension corresponds to the size of a minibatch. Attributes: start_transitions (`~torch.nn.Parameter`): Start transition score tensor of size ``(num_tags,)``. end_transitions (`~torch.nn.Parameter`): End transition score tensor of size ``(num_tags,)``. transitions (`~torch.nn.Parameter`): Transition score tensor of size ``(num_tags, num_tags)``. .. [LMP01] Lafferty, J., McCallum, A., Pereira, F. (2001). "Conditional random fields: Probabilistic models for segmenting and labeling sequence data". *Proc. 18th International Conf. on Machine Learning*. Morgan Kaufmann. pp. 282–289. .. _Viterbi algorithm: https://en.wikipedia.org/wiki/Viterbi_algorithm Fnum_tagsrreturnNcsp|dkr td|t||_||_tt||_ tt||_ tt|||_ | dS)Nrzinvalid number of tags: ) ValueErrorrrrTrr Parameterr8emptystart_transitionsend_transitions transitionsreset_parameters)r!rTrr#r%r&rs  z CRF.__init__cCs:tj|jddtj|jddtj|jdddS)zInitialize the transition parameters. The parameters will be initialized randomly from a uniform distribution between -0.1 and 0.1. gg?N)rinituniform_rYrZr[r!r%r%r&r\szCRF.reset_parameterscCs|jjd|jdS)Nz (num_tags=))r$rDrTr_r%r%r&__repr__sz CRF.__repr__mean emissionstagsr? reductioncCs|dvr td||durtj|tj|jd}|jtjkr#|}|j|||d|jr@| dd}| dd}| dd}| |||}| ||}||}|dkrW|S|d kr_| S|d krg| S| | S) aCompute the conditional log likelihood of a sequence of tags given emission scores. Args: emissions (`~torch.Tensor`): Emission score tensor of size ``(seq_length, batch_size, num_tags)`` if ``batch_first`` is ``False``, ``(batch_size, seq_length, num_tags)`` otherwise. tags (`~torch.LongTensor`): Sequence of tags tensor of size ``(seq_length, batch_size)`` if ``batch_first`` is ``False``, ``(batch_size, seq_length)`` otherwise. mask (`~torch.ByteTensor`): Mask tensor of size ``(seq_length, batch_size)`` if ``batch_first`` is ``False``, ``(batch_size, seq_length)`` otherwise. reduction: Specifies the reduction to apply to the output: ``none|sum|mean|token_mean``. ``none``: no reduction will be applied. ``sum``: the output will be summed over batches. ``mean``: the output will be averaged over batches. ``token_mean``: the output will be averaged over tokens. Returns: `~torch.Tensor`: The log likelihood. This will have size ``(batch_size,)`` if reduction is ``none``, ``()`` otherwise. )noner6rb token_meanzinvalid reduction: Ndtyper4)rdr?rr3rfr6rb)rVr8 ones_likeuint8r4ribyte _validater transpose_compute_score_compute_normalizerr6rbfloat)r!rcrdr?re numerator denominatorllhr%r%r&r/s*     z CRF.forwardnbestpad_tagcCs|durd}|durtj|jddtj|jd}|jtjkr#|}|j||d|jr9| dd}| dd}|dkrG| ||| dS| ||||S)aFind the most likely tag sequence using Viterbi algorithm. Args: emissions (`~torch.Tensor`): Emission score tensor of size ``(seq_length, batch_size, num_tags)`` if ``batch_first`` is ``False``, ``(batch_size, seq_length, num_tags)`` otherwise. mask (`~torch.ByteTensor`): Mask tensor of size ``(seq_length, batch_size)`` if ``batch_first`` is ``False``, ``(batch_size, seq_length)`` otherwise. nbest (`int`): Number of most probable paths for each sequence pad_tag (`int`): Tag at padded positions. Often input varies in length and the length will be padded to the maximum length in the batch. Tags at the padded positions will be assigned with a padding tag, i.e. `pad_tag` Returns: A PyTorch tensor of the best tag sequence for each batch of shape (nbest, batch_size, seq_length) Nr3rrh)r?r) r8onesr:rkr4rirlrmrrn_viterbi_decoderO_viterbi_decode_nbest)r!rcr?rurvr%r%r&r;s"    z CRF.decodecCs|dkrtd||d|jkr%td|jd|d|durG|jdd|jkrGtdt|jdddt|j|dur|jdd|jkritdt|jdddt|j|j or|d }|jo|ddd f}|s|std dSdSdS) Nz(emissions must have dimension of 3, got rz(expected last dimension of emissions is z, got z?the first two dimensions of emissions and tags must match, got z and z?the first two dimensions of emissions and mask must match, got rz)mask of the first timestep must all be on)dimrVsizerTr:tuplerall)r!rcrdr? no_empty_seqno_empty_seq_bfr%r%r&rmsD   z CRF._validatec Cs|j\}}|}|j|d}||dt||df7}td|D]'}||j||d||f||7}|||t|||f||7}q#|jddd}||t|f} ||j | 7}|S)Nrr3r{) r:rqrYr8r9rMr[r7r6rZ) r!rcrdr? seq_length batch_sizescorerSseq_ends last_tagsr%r%r&ro"s &&zCRF._compute_scorec Cs|d}|j|d}td|D](}|d}||d}||j|}tj|dd}t||d||}q||j7}tj|ddS)Nrr3rr) r|rYrMrOr[r8 logsumexpwhererZ) r!rcr?rrrSbroadcast_scorebroadcast_emissions next_scorer%r%r&rpBs   zCRF._compute_normalizerc Cs|durd}|j}|j\}}|j|d}tj|||jftj|d}tj||jftj|d} tj||f|tj|d} td|D]?} | d} ||  d} | |j | }|j dd\}}t ||  d ||}t ||  d || }||| d<q?||j}|j dd\}}|jddd}|dd}|d|ddddd|j|ddddd|j|dd}tj||ftj|d}tj|dtj|d}t|dddD]}t||d|}|j|||<qt | || ddS)Nrrhr3rrr2)r4r:rYr8zerosrTr7fullrMrOr[maxrboolrZr6rn contiguousscatter_rPexpandgatherdata)r!rcr?rvr4rrr history_idxoor_idxoor_tagrSrbroadcast_emissionrindices end_score_end_tagr best_tags_arr best_tagsidxr%r%r&rxpsb      zCRF._viterbi_decodec Cs|durd}|j}|j\}}|j|d}tj|||j|ftj|d} tj||j|ftj|d} tj|||f|tj|d} td|D]} | dkr\| d} ||  d}| |j |}n| d} ||  d d}| |j d|}| |d|jj |dd\}}| dkr| d dd|}||}|dd}|dd}t||  d d||}t||  d d|| }|| | d<qB||j d}| |dj |dd\}}|jddd}| dd} | d| dddd dd|j|| ddd| dd|j|| dd} tj|||ftj|d}tj|tj|d dd |d}t|dddD]}t| | |dd|}|j |d|||<q@t| d|| dddS)Nrrhr3r2rr)r4r:rYr8rrTr7rrMrOr[rPtopkrrnrrrZr6rrr9rrpermute)r!rcr?rurvr4rrrrrrrSrrrrrrrrrrrr%r%r&rys           zCRF._viterbi_decode_nbest)F)rUN)Nrb)NNN)NN)N)rDrErF__doc__intrrr\strrar8Tensor LongTensorr ByteTensorr/rr;rmrorp FloatTensorrxryrGr%r%r#r&rzs   5 )    1  ar)&typingrrrrr8torch.nn.functionalr functionalFtorch.nnrtransformers.activationsrmodelscope.metainfor modelscope.models.baser modelscope.models.builderr modelscope.outputsr r rrmodelscope.utils.constantrregister_moduletoken_classificationlstm_crfnamed_entity_recognitionword_segmentationpart_of_speechrtransformer_crfrHModulerr%r%r%r&sB       *