o iE@sRddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl m Zddl mZmZddlmZddlmZddlmZmZmZddlmZdd lmZmZmZdd lm Z dd l!m"Z"d d l#m$Z$d dl%m&Z&m'Z'd dl(m)Z)dZ*e+e*Z,dZ-ej.ddge,ddddddede/de de0de0ddf dd Z1Gd!ddeZ2d%d#d$Z3d"S)&)islice) AnyCallableDictIterableIteratorListOptionalSequenceTuple)Errors)LanguageVocab) TrainablePipe)Doc)Examplevalidate_examplesvalidate_get_examples) minibatch)Config Optimizerset_dropout_rate)Model)Ragged)ListenerStateUtils)DocTransformerOutputTransformerModelOutput)TransformerListenerModelTah [transformer] [transformer.model] @architectures = "spacy-curated-transformers.XlmrTransformer.v1" vocab_size = 250002 [transformer.model.piece_encoder] @architectures = "spacy-curated-transformers.XlmrSentencepieceEncoder.v1" [transformer.model.with_spans] @architectures = "spacy-curated-transformers.WithStridedSpans.v1" trf_datacurated_transformerzdoc._.trf_data transformer)assignsdefault_configF)frozenall_layer_outputsnlpnamemodelr$r%returnCuratedTransformercCst|j||||dS)auConstruct a CuratedTransformer component, which lets you plug a pre-trained transformer model into spaCy so you can use it in your pipeline. One or more subsequent spaCy components can use the transformer outputs as features in its model, with gradients backpropagated to the single shared weights. nlp (Language): The pipeline. name (str): The component instance name. model (Model): One of the supported pre-trained transformer models. frozen (bool): If `True`, the model's weights are frozen and no backpropagation is performed. all_layer_outputs (bool): If `True`, the model returns the outputs of all the layers. Otherwise, only the output of the last layer is returned. This must be set to `True` if any of the pipe's downstream listeners require the outputs of all transformer layers. r'r$r%)r*vocab)r&r'r(r$r%r-c/home/ubuntu/.local/lib/python3.10/site-packages/spacy_curated_transformers/pipeline/transformer.pymake_transformer/sr/c @seZdZdZdddddedededed ed d f d d Ze d e e fddZ e d e efddZ de ded d fddZde ded efddZded d fddZdddeeded eefdd Zd!eed efd"d#Zd!eed$ed d fd%d&Zd'd d d(d)eed*ed+eed,eeeefd eeeff d-d.Z d)eed/ed d fd0d1Z!d d d d2d3e"geefd4ee#d5ee"d6ee"fd7d8Z$d9efd:d;Z%d+ed d fdd!eed,eeefd+eed e'ee"e"ffd?d@Z(dAefdBdCZ)d S)Dr*auspaCy pipeline component that provides access to a pre-trained transformer model from. Downstream components are connected to this pip using transformer listener layers. This works similarly to spaCy's Transformer component and TransformerListener sublayer. The activations from the transformer are saved in the doc._.trf_data extension attribute. r!Fr+r,r(r'r$r%r)NcCs>||_||_||_i|_i|_t||_||_||dS)az vocab (Vocab): The shared vocabulary. model (Model): One of the supported pre-trained transformer models. name (str): The component instance name. frozen (bool): If `True`, the model's weights are frozen and no backpropagation is performed. all_layer_outputs (bool): If `True`, the model returns the outputs of all the layers. Otherwise, only the output of the last layer is returned. This must be set to `True` if any of the pipe's downstream listeners require the outputs of all transformer layers. N) r,r(r' listener_mapcfg_install_extensionsr$r%_set_model_all_layer_outputs)selfr,r(r'r$r%r-r-r.__init__aszCuratedTransformer.__init__csfddjDS)z RETURNS (List[TransformerListenerModelT]): The listener models listening to this component. Usually internals. cs g|] }j|D]}|q qSr-)r0).0cmr4r-r. s z0CuratedTransformer.listeners..)listening_componentsr9r-r9r. listenersszCuratedTransformer.listenerscCst|jS)z| RETURNS (List[str]): The downstream components listening to this component. Usually internals. )listr0keysr9r-r-r.r;sz'CuratedTransformer.listening_componentslistenercomponent_namecCsPt|std|jd|j|g||j|vr&|j||dSdS)z=Add a listener for a downstream component. Usually internals.z!Attempting to register a model ('z=') with the transformer pipethat isn't a transformer listenerN)r is_listener ValueErrorr'r0 setdefaultappendr4r?r@r-r-r. add_listeners  zCuratedTransformer.add_listenercCsB||jvr||j|vr|j|||j|s|j|=dSdS)zRemove a listener for a downstream component. Usually internals. RETURNS (bool): `True` if successful, `False` otherwise. TF)r0removerEr-r-r.remove_listeners  z"CuratedTransformer.remove_listener componentcCsZd|jf}tt|ddtr)|jD]}t|r(t||vr(| ||jqdSdS)aWalk over a model of a processing component, looking for layers that are transformer listeners that have an upstream_name that matches this component. Listeners can also set their upstream_name attribute to the wildcard string '*' to match any `Transformer`. You're unlikely to ever need multiple `Transformer` components, so it's fine to leave your listeners upstream_name on '*'. *r(N) r' isinstancegetattrrr(walkrrAget_upstream_namerF)r4rInamesnoder-r-r.find_listenerss z!CuratedTransformer.find_listeners) batch_sizestreamrSccs<tt||D]}||}||||EdHq dS)aApply the pipe to a stream of documents. This usually happens under the hood when the nlp object is called on a text and all components are applied to the Doc. stream (Iterable[Doc]): A stream of documents. batch_size (int): The number of documents to buffer. YIELDS (Doc): Processed documents in order. N)r2rpredictset_annotations)r4rTrSbatchpredsr-r-r.pipes    zCuratedTransformer.pipedocscsXttdd|Ds jdtfdd|DddSjj|S)zApply the pipeline's model to a batch of docs, without modifying them. docs (Iterable[Doc]): The documents to predict. RETURNS (TransformerModelOutput): The extracted features of each document. css|]}t|VqdSN)len)r6docr-r-r. sz-CuratedTransformer.predict..nOcs0g|]}tjjdjjddgqS)r)datalengths)rr(opsalloc2falloc1i)r6_r4widthr-r.r:s z.CuratedTransformer.predict..T)outputslast_layer_only)r2anyr(get_dimrr3r%rU)r4rZr-rfr.rUs      zCuratedTransformer.predict trf_outputcCs,t||jD] \}}t||jd|j_qdS)a;Assign the extracted features to the Doc objects. By default, a DocTransformerOutput object is written to the doc._.trf_data attribute. docs (Iterable[Doc]): The documents to modify. trf_output (TransformerModelOutput): The outputs of the transformer model. ) all_outputsriN)ziprmrrirer)r4rZrlr]tokvecsr-r-r.rVs   z"CuratedTransformer.set_annotations)dropsgdlossesexamplesrqrrrsc Cs|duri}t|ddd|D}t|j|||jd||j|j|||d\}}}t |} |j ddD] } t | | ||q<|j rUt |j d| |||S)aPrepare for an update to the transformer. Like the `Tok2Vec` component, the `Transformer` component is unusual in that it does not receive "gold standard" annotations to calculate a weight update. The optimal output of the transformer data is unknown; it's a hidden layer inside the network that is updated by backpropagating from output layers. The `Transformer` component therefore does not perform a weight update during its own `update` method. Instead, it runs its transformer model and communicates the output and the backpropagation callback to any downstream components that have been connected to it via the transformer listener sublayer. If there are multiple listeners, the last layer will actually backprop to the transformer and call the optimizer, while the others simply increment the gradients. examples (Iterable[Example]): A batch of Example objects. Only the `predicted` doc object is used, the reference doc is ignored. drop (float): The dropout rate. sgd (thinc.api.Optimizer): The optimizer. losses (Dict[str, float]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. NzTransformer.updatecSsg|]}|jqSr-) predicted)r6egr-r-r.r:-sz-CuratedTransformer.update..rprr) rrr(rCr'r3r%_create_backpropsrcalculate_batch_idr<receive) r4rtrqrrrsrZrh accum_func backprop_funcbatch_idr?r-r-r.updates$&    zCuratedTransformer.updatescorescCsdS)zA noop function, for compatibility with the Pipe API. See the `update` method for an explanation of the loss mechanics of the component. Nr-)r4rtrr-r-r.get_lossAszCuratedTransformer.get_loss)r&encoder_loader piecer_loader get_examplesr&rrcCszt|d|r||jd_|r||jd_g}t|dD]}||jq|s4Jtjj |j d|jj |ddS)aInitialize the pipe for training, using data examples if available. get_examples (Callable[[], Iterable[Example]]): Optional function that returns gold-standard Example objects. nlp (Language): The current nlp object. encoder_loader (Optional[Callable]): Initialization callback for the transformer model. piecer_loader (Optional[Callable]): Initialization callback for the input piece encoder. zTransformer.initializer! piece_encoder )r')XN) rr(get_refinitrrDx SpacyErrorsE923formatr' initialize)r4rr&rr doc_sampleexampler-r-r.rGs zCuratedTransformer.initializelabelcCstr[)NotImplementedError)r4rr-r-r. add_labelgszCuratedTransformer.add_labelcCs|js |j|dSdS)aQUpdate parameters using the current parameter gradients. The Optimizer instance contains the functionality to perform the stochastic gradient descent. This method is a noop when the pipe is frozen. sgd (thinc.api.Optimizer): The optimizer. DOCS: https://spacy.io/api/pipe#finish_update N)r$r( finish_update)r4rrr-r-r.rjs z CuratedTransformer.finish_updaterwcsjjjdjjrj|}dnj|\}fdd|jDdt t t dt t dfddffdd fd d dt t t dt t dfdt ffd d }dt t t dt t dfdt ffdd }|jr|fS|fS)Ncsg|] }fdd|DqS)cs"g|] }t|jj|jqSr-)ralloc_fdataXdshapera)r6t2vrbr-r.r:s"zCCuratedTransformer._create_backprops...r-)r6 doc_layersrr-r.r:sz8CuratedTransformer._create_backprops.. one_d_outputsoutputs_to_backprop.r)cs@tt|D]}|D]}||j|||j7_q qdS)a.Accumulate transformer loss and gradient. This is passed as a callback to all but the last listener. Only the last one does the backprop. `outputs_to_backprop` is a tuple of indices indicating to which layers/outputs the gradients are to be propagated. N)ranger\r`)rrij) d_outputsr-r.accumulate_gradients  "zACuratedTransformer._create_backprops..accumulate_gradientcsTttD]}tt|D]}||jd7qqtj<dS)z6Reduce the gradient buffer and update the losses dict.rN)rr\r`sumfloatr')rr)cum_lossrrsr4r-r. update_losss z9CuratedTransformer._create_backprops..update_losscs<dusJ||d}dur|S)z>Callback to actually do the backprop. Passed to last listener.Nrr)rrd_docs)r bp_outputsrr4rrrr-r.backprops   z6CuratedTransformer._create_backprops..backpropcs(||ddurgS)Nrr)rr)rr4rrrr-r. backprop_noops  z;CuratedTransformer._create_backprops..backprop_noop)r(rbxpfullr'r$rU begin_updatermrrr intr)r4rZrsrrrhrrr-) rrrrrsrbr4rrrr.ryxs@         z$CuratedTransformer._create_backprops new_valuecCs||jdjd<dS)Nr!_all_layer_outputs)r(rattrs)r4rr-r-r.r3sz/CuratedTransformer._set_model_all_layer_outputs)*__name__ __module__ __qualname____doc__rrstrboolr5propertyrrr<r;rFrHrrQrrrrrYrrUr rVrrr rrrrrr rrrr ryr3r-r-r-r.r*Ws "  $   =      ONcCs ttstjtdddSdS)N)default)r has_extension DOC_EXT_ATTR set_extensionr-r-r-r.r2s r2)r)N)4 itertoolsrtypingrrrrrrr r r spacyr rr rspacy.pipeliner spacy.tokensrspacy.trainingrrr spacy.utilr thinc.apirrr thinc.modelr thinc.typesrmodels.listenersr models.outputrr models.typesrDEFAULT_CONFIG_STRfrom_strDEFAULT_CONFIGrfactoryrrr/r*r2r-r-r-r.sR ,           #v