o }oi6 @sddlZddlZddlmZmZddlZddlmmZddlmZddl m Z ddl m Z m Z ddlmZmZddlmZz ddlmZmZd ZWneefyZe Zd ZYnwd d gZGd d d ejZGdddeeZGdddeeZGdddeeZGdd d eeZdS)N)DictOptional)nn)fused_bias_gelu)ApexGuardDefaultsinit_method_normal) Exportable NeuralModule) typecheck)ModelParallelConfigtensor_parallelTF PromptEncoderPromptEncoderTypec@seZdZdZdZdZdZdS)rtpmlpmlplstm embeddingN)__name__ __module__ __qualname__TPMLPMLPLSTM EMBEDDINGrrf/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/prompt_encoder.pyr+s cs@eZdZdZfddZddZdejfddZd d Z Z S) PromptEmbeddinga|Prompt embeddings Arugments: init_from_prompt_text: Whether to intialize prompt embeddings from from certain lm embeddings corresponding to a prompt string hidden_size: hidden size should match lm embedding size total_virtual_tokens: length of prompt initalized from torch init method csht||_||_tj|j|j|_|jjj dd|jj_ |j dt tt|jdddS)NFindices) persistent)super__init__ hidden_sizetotal_virtual_tokenstorchr Embeddingprompt_embeddingsweightdatafill_ requires_gradregister_buffer LongTensorlistrange)selfr"r# __class__rrr!=s  $zPromptEmbedding.__init__cCs|jjddS)zA Method sets the prompt embedding weights to 0.0 rN)r&r'r)r/rrrclear_prompt_embedding_weightsMsz.PromptEmbedding.clear_prompt_embedding_weightsr'cCs||jjj|jj_dS)zN Method sets the prompt embedding weights with a new weight w N)type_asr&r'r()r/r'rrrset_prompt_embedding_weightsSsz,PromptEmbedding.set_prompt_embedding_weightscCs ||jS)z$ Does forward pass )r&rr2rrrforwardYs zPromptEmbedding.forward) rrr__doc__r!r3r$Tensorr5r6 __classcell__rrr0rr2s  rcsBeZdZdZd fdd ZdejfddZdd Zd d Z Z S) InferenceTablez A wrapper class that holds the output representations of the PromptEncoder Model. At inference time we do not need to forward pass through the full PromptEncoder and can just use this class. Fcsnt||_||_||_tj|_t |j|j|j|j<|j|j ||_ |j D]}d|_ q/dS)NF)r r!tasknamer"r#r$r ModuleDict prompt_tablerr3is_inference_ready parametersr*)r/r;r"r#r>pr0rrr!fs  zInferenceTable.__init__prompt_representationcCs|j|j|d|_dS)zf Method sets the prompt embedding inside self.prompt_table[taskname] with new weights TN)r=r;r5r>r/rArrrset_prompt_tablers zInferenceTable.set_prompt_tablecCs|j|jS)zO Returns the prompt representation cached in the prompt table )r=r;r6r2rrrget_prompt_tableyszInferenceTable.get_prompt_tablecCs|j|jd|_dS)zp Method "clears" the prompt embedding inside self.prompt_table[taskname] by setting it to zero. FN)r=r;r3r>r2rrrclear_prompt_tables z!InferenceTable.clear_prompt_table)F) rrrr7r!r$r8rCrDrEr9rrr0rr:`s  r:c sFeZdZdZdededededef fdd Zd ej fd d Z Z S) rz The Tensor Parallel MLP prompt encoder network that is used to generate the virtual token embeddings for p-tuning. It only have two layers. configr#r" output_sizeinit_stdc st||_||_||_d|_t|}d|_d|_ t j |j|j|dt |ddd|_ t j|j|j|dt |ddd|_dS)ar Initializes the Tensor Model parallel MLP PromptEncoderMLP module. Args: config: the model parallel config used my megatron core total_virtual_tokens: the total number of vitural tokens hidden_size: hidden dimension output_size: the output dimension init_std: the MLP init std value geluFT)rF gather_output init_method skip_bias_addbias)rFinput_is_parallelrKrLrMN)r r!r"rGr# activationcopydeepcopysequence_parallelgradient_accumulation_fusionr ColumnParallelLinearrfirstRowParallelLinearsecond)r/rFr#r"rGrHr0rrr!s4    zTPMLP.__init__returncCs2||\}}t||}||\}}||}|SN)rUrrW)r/ input_embedsintermediate_parallel bias_parallel output_embedsrrrr6s  z TPMLP.forward) rrrr7r intfloatr!r$r8r6r9rrr0rrs.rcseZdZdZ d!dedededededed edeffd d Z d e j fd dZ ddZ ddZd"ddZd#ddZddZededede j fdd ZZS)$r zm The prompt encoder network that is used to generate the virtual token embeddings for p-tuning. r;rF encoder_typer# token_dim lstm_dropout num_layersrHc st||_||_||_||_||_||_d|_||_ | |_ | dt tt|jt j|j|j|_t| |j|j|_|jtjkrPt|jjd S|jtjkrt jj|j|j||ddd|_tt|jd|jdtt|jd|j|_d S|jtj kr|dkrt!dt|j|jtg} t|dD]} | "t|j|jtgq| #t|j|jtj| |_d S|jtj$krt$||j|j|j|j |_%d St!d) a Initializes the PromptEncoder module. Args: config: the model parallel config used my megatron core total_virtual_tokens: the total number of vitural tokens hidden_size: hidden dimension lstm_dropout: the dropout used for the LSTM num_layers: number of layers used in the LSTM init_std: used for TPMLP encoder type to initialize the mlp weights rIrT) input_sizer"rcdropout bidirectional batch_firstzXThe MLP prompt encoder must have at least 2 layers, and exactly 2 layers is recommended.zPPrompt encoder type not recognized. Please use one of MLP (recommended) or LSTM.N)&r r!rardrGr"r#r`rOrHr;r+r$r,r-r.rr%rr:inference_tablerrinitxavier_normal_r'r lstm_head SequentialLinearReLUmlp_headr ValueErrorextendappendrr) r/rFr`r#rar"rbrcrHr;layers_r0rrr!sT       zPromptEncoder.__init__rAcCs|}|j|dS)z{ This method caches the output representation from the Encoder and saves it inside `self.inference_table`. N)detachclonerjrCrBrrrset_inference_tables z!PromptEncoder.set_inference_tablecCs|jdSrY)rjrEr2rrrclear_inference_tablesz#PromptEncoder.clear_inference_tablecCs |jSrY)rjrDr2rrrget_inference_table s z!PromptEncoder.get_inference_tableNFcCsi}|j|d<|j|d<|jtjkr |S|jtjkr/|j|d<|j|d<|S|jtj kr>|j|d<|S|jtj krM|j |d<|St dNr= embeddingsrqrmr'Prompt encoder type not recognized. Pl.) rj state_dictrr`rrrrqrmrrrrr)r/ desinationprefix keep_vars _state_dictrrrr#s(    zPromptEncoder.state_dictTcCs|j|d|j|d|jtjkr dS|jtjkr1|j|d|j |ddS|jtj krA|j|ddS|jtj krQ|j |ddSt dr|)rjload_state_dictrr`rrrrqrmrrrrrr)r/rstrictrrrr8s     zPromptEncoder.load_state_dictcCs||jd}|jtjkr|}|S|jtjkr%|||d}|S|jtj kr2||}|S|jtj kr?| |}|St d)Nrr~) rr unsqueezer`rrrrqrmrrrrr)r/rZr]rrr_forwardHs    zPromptEncoder._forward batch_sizeuse_cached_repsrXcCsx|r |d}n'|jr|jjr||}n|jjs*|}||d|d}| ||j |j }|S)zZ Forward pass through the encoder with caching of prompt representations r) r{rtrainingrjr>rzrrysqueezeexpandr#ra)r/rrr]rrrr6Vs zPromptEncoder.forward)r;)NNF)T)rrrr7r enumr^r_strr!r$r8ryrzr{rrrr boolr6r9rrr0rr s8  M  ") rPrtypingrrr$ torch.nn.initrrks.   .'<