o ei L@s&dZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZdd lmZdd lmZdd lmZmZdd lmZddlmZmZmZddlmZmZmZm Z ddl!m"Z"e #e$Z%  dvdej&dej'dej'dej'dej'dBde(dBde(deefddZ)Gdddej&Z*Gdd d ej&Z+  ! dwd"ej'd#e(d$e,dBd%e-d&e.f d'd(Z/  dxd"ej'd)e,e.Bd$e,dBd&e.fd*d+Z0Gd,d-d-ej&Z1Gd.d/d/ej&Z2Gd0d1d1ej&Z3eGd2d3d3eZ4Gd4d5d5ej&Z5Gd6d7d7ej&Z6Gd8d9d9e4Z7eed:d;Gdd;Gd?d@d@eZ9eedAd;GdBdCdCeZ:eedDd;GdEdFdFeZ;eedGd;GdHdIdIeZj?dNej'dOej'fdPdQZ@dydRej'dSej'dBdOej'fdTdUZAGdVdWdWej&ZBGdXdYdYej&ZCGdZd[d[ej&ZDGd\d]d]ej&ZEeGd^d_d_e4ZFGd`dadaej&ZGedbd;Gdcdddde4ZHGdedfdfej&ZIedgd;Gdhdidie4ZJedjd;Gdkdldlej&ZKedmd;Gdndodoe4ZLGdpdqdqej&ZMedrd;Gdsdtdte4ZNgduZOdS)zzPyTorch PatchTST model.N)Callable) dataclass)nn)initialization)ACT2CLS)is_deepspeed_zero3_enabled)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputTransformersKwargsauto_docstringlogging)PatchTSTConfigmodulequerykeyvalueattention_maskscalingdropoutkwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nrdim)ptrainingr) sizetorchmatmul transposer functionalsoftmaxrr& contiguous) rrrrrrrr attn_weights attn_outputr0l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/patchtst/modeling_patchtst.pyeager_attention_forward's  r2cseZdZdZ     ddededed ed ed ed edBffd d Z   dde j de j dBde j dBdedBde e de e j e j dBe e j dBff ddZZS)PatchTSTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderbias is_causalconfigcst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r!r7)super__init__r4r5rhead_dimr9 ValueErrorrr6r8rLineark_projv_projq_projout_proj)selfr4r5rr6r7r8r9 __class__r0r1r=Gs&    zPatchTSTAttention.__init__ hidden_stateskey_value_statesroutput_attentionsrreturncKs|du}|jdd\}}|r|jdn|} ||d|jf} || d|jf} ||j| dd} |r4|n|} || j| dd}|| j| dd}t|j j t }||| |||f|j sbdn|j |j|d|\}}|||d}||}||dfS)z#Input shape: Batch x Time x ChannelNr rr"r)rrrJ)shaper>rCviewr*rArBr get_interfacer9_attn_implementationr2r&rrreshaper-rD)rErHrIrrJris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacer/r.r0r0r1forwardfs8    zPatchTSTAttention.forward)rFTFN)NNF)__name__ __module__ __qualname____doc__intfloatboolrr=r(Tensorr r tupler\ __classcell__r0r0rFr1r3DsL" r3c6eZdZdZdeffdd ZdejfddZZ S)PatchTSTBatchNormzP Compute batch normalization over the sequence length (time) dimension. r9cs"ttj|j|jd|_dS)Neps)r<r=r BatchNorm1dd_modelnorm_eps batchnormrEr9rFr0r1r=s zPatchTSTBatchNorm.__init__inputscCs"|dd}||}|ddS)a Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` rr")r*rn)rErpoutputr0r0r1r\s   zPatchTSTBatchNorm.forward r]r^r_r`rr=r(rdr\rfr0r0rFr1rhsrhFrp mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuecCs*|dks|dkrtd|d|j\}}}}|j} t|d|} |r5tj|d|| d} | d|d} n tj|||| d} tj|||| d} d| ddddd| f<tj| dd} tj| dd}tj | d|d } | dddd|} |durd| dd|ddddf<| | |}|| d fS) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNr r#)r$index.r) r?rLrxrar(randrepeatonesargsortgather unsqueeze masked_fillrc)rprsrtrurv batch_size num_channelssequence_length num_featuresrxlen_keepnoisemask ids_shuffle ids_restore inputs_maskr0r0r1random_maskings& rnum_forecast_mask_patchescCst|tr|g}dd|D}|j\}}}}tj||||jd} g} d} t|} t||D](\} }| dks9| |krAtd| dt||| }| | ||g| |7} q-t | ddd } | |krq| dd || | dd <n| |kr| d d | || d d <d}| D]\}}}||}d | ||d d | d f<|}qt | jd}| |} | d  d d d |} |d urd| d d |d d d d f<|| |}|| dfS)aForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` cSsg|]}dqS)rr0.0_r0r0r1 sz$forecast_masking..rwrznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.cSs|dS)Nr"r0)xr0r0r1sz"forecast_masking..)rr"r rNrz) isinstancerarLr(zerosrxsumzipr?appendsortedrandpermrr|rrc)rprrtrvforecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenrbatch2permrr0r0r1forecast_maskingsB    rcrg)PatchTSTPatchifyz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r9cst|j|_|j|_|j|_|j|jkr$td|jd|jdt|j|j|j|jd|_|j|j|jd}|j||_ dS)NzSequence length (z+) has to be greater than the patch length ()r) r<r=context_lengthrr patch_strider?max num_patchessequence_start)rEr9new_sequence_lengthrFr0r1r=6s   zPatchTSTPatchify.__init__ past_valuescCsp|jd}||jkrtd|d|jd|dd|jdddf}|jd|j|jd}|dd}|S)a! Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (r:N) dimensionr'step) rLrr?runfoldrrr*r-)rErrrqr0r0r1r\Gs zPatchTSTPatchify.forwardrrr0r0rFr1r.srcrg)PatchTSTMaskinga Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r9csXt|j|_|j|_|j|_|j|_|j|_|j|_|jdur*t|j|_dSdSN) r<r=random_mask_ratioru mask_typerrtrvrrorFr0r1r=ks  zPatchTSTMasking.__init__ patch_inputcCsr|jdkrt||j|j|j|jd\}}n|jdkr(t||j|j|jd\}}n td|jd| }||fS)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)rprsrtrurvforecast)rprrtrvzInvalid mask type .) rrrrtrurvrrr?rc)rEr masked_inputrr0r0r1r\vs$   zPatchTSTMasking.forwardrrr0r0rFr1r^s  rcs@eZdZdZdeffdd Zd dejdedBfdd Z Z S) PatchTSTEncoderLayerz PatchTST encoder layer r9c st|j|_t|j|j|j|d|_|jdkr t |jnt |_ |j dkr0t||_n|j dkr@t j|j|jd|_nt|j d|jr~|jdkrVt |jnt |_|j dkrft||_n|j dkrvt j|j|jd|_nt|j dt t j|j|j|jdt|j|jdkrt |jnt t j|j|j|jd|_|jdkrt |jnt |_|j dkrt||_n|j dkrt j|j|jd|_nt|j d|j|_dS)N)r4r5rr9rrn layernormriz$ is not a supported norm layer type.r;) r<r=channel_attentionr3rlnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typerhnorm_sublayer1 LayerNormrmr? dropout_path2norm_sublayer2 Sequentialr@ffn_dimr7ractivation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrorFr0r1r=sD               zPatchTSTEncoderLayer.__init__N hidden_staterJc Cs|j\}}}}|||||}|jr(|j|||d\}}} |||}n|j||d\}}} ||||}|||||}|jr|dd }|||||}|jrp|j| ||d\}} } || |}n|j||d\}} } | || |}|||||}|dd }|||||}|jr|| | ||}n ||| | |}|||||}|f} |r| |jr|| fn|f7} | S)a Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )rHrJr"r)rLrMrrrrrPrr*r-rrrrr) rErrJrnum_input_channelsrrlr/r.rchannel_attn_weightsoutputsr0r0r1r\sF       zPatchTSTEncoderLayer.forwardr) r]r^r_r`rr=r(rdrcr\rfr0r0rFr1rs"2rc@sTeZdZUeed<dZdZdZdZdZ dZ dZ e dejfdd Zd d d Zd S)PatchTSTPreTrainedModelr9modelr)timeFTrcCs~t|trjt|jj|jj|jj|jjd}|jjr(tj |j dd|d7}| |j|}t raddl }|jj|jdd|jdkrNt|j|Wdn 1sXwYdSdSt|j|dSt|tjtjfrt|jt|jt|dddurt|jt|jt|jdSdSt|tjrtj |jd|jjd |jdurt|jdSdSdS) z$ Initialize weights rg{Gz?)stdrN) modifier_rank running_meanr)meanr) rPatchTSTPositionalEncodingrr9rrr use_cls_tokeninitnormal_ cls_token_init_per deepspeedzeroGatheredParameters position_encnumelcopy_rrrkzeros_r7ones_weightgetattrr running_varnum_batches_trackedr@init_std)rErrrrr0r0r1 _init_weights0s@ $      z%PatchTSTPreTrainedModel._init_weightscCst|tr ||_dSdSr)rPatchTSTEncodergradient_checkpointing)rErrr0r0r1_set_gradient_checkpointingTs  z3PatchTSTPreTrainedModel._set_gradient_checkpointingN)F)r]r^r_r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr(no_gradrModulerrr0r0r0r1r%s #rc2eZdZdeffdd ZdejfddZZS)PatchTSTEmbeddingr9cslt|j|_|j|_|jrt|j|j|_dSt |_t |jD]}|j t|j|jq%dSr) r<r=rshare_embeddingrr@rrlinput_embedding ModuleListranger)rEr9rrFr0r1r=Zs  zPatchTSTEmbedding.__init__rcsjjd}|jkrtdjd|djr }|Sfddt|D}tj|dd}|S)a% Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` rz&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rc s2g|]}j|dd|ddddfqSr)rrirrEr0r1rxs2z-PatchTSTEmbedding.forward..r#)rLrr?rrrr(stack)rErr embeddingsr0r r1r\fs   zPatchTSTEmbedding.forward r]r^r_rr=r(rdr\rfr0r0rFr1rYs rcsVeZdZdZdedeffdd Zedededej fddZ d e j fd d Z ZS) rz' Class for positional encoding r9rcszt|j|_|j|_|jr!ttddd|j|_ |d7}| |||_ |j dkr6t |j |_ dSt|_ dS)Nrr)r<r=rrr Parameterr(rrlrrrpositional_dropoutrrrEr9rrFr0r1r=s z#PatchTSTPositionalEncoding.__init__rKcCs|jdkrtjt||jdd}|S|jdkrst||j}td|d}t td|jdt d|j }t |||dddddf<t |||dddddf<||}||d }tj|d d}|St|jd ) NrT requires_gradsincosrrr"g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerr r(randnrlrarangerexpmathlogsincosrrr?)r9rrpositiondiv_termr0r0r1rs  (    z#PatchTSTPositionalEncoding._init_percCs|jr8|||jddddf}|j|jddddf}||jd|jdd}tj||fdd}|S|||j}|S)Nrrr r"r#) rrrrexpandrLrr(cat)rErr cls_tokensrr0r0r1r\s z"PatchTSTPositionalEncoding.forward)r]r^r_r`rrar= staticmethodrr rr(rdr\rfr0r0rFr1r}s rc sTeZdZdZdedeffdd Z  d dejde dBd e dBd e fd d Z Z S)rz PatchTST Encoder r9rcsTtd|_t|_t||_tfddt j D|_ | dS)NFcsg|]}tqSr0)rrr9r0r1rz,PatchTSTEncoder.__init__..) r<r=rrembedderrpositional_encoderrrrnum_hidden_layerslayers post_initrrFr"r1r=s     zPatchTSTEncoder.__init__Nroutput_hidden_statesrJrKc Ks|dur|n|jj}|dur|n|jj}||}||}|r"dnd}|r(dnd}|jD]}|r6||f}|||d} | d}|rI|| df}q-t|||dS)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Past values of the time series output_hidden_states (bool, optional): Indicates if hidden states should be outputted. output_attentions (bool, optional): Indicates if attentions should be outputted. return: `BaseModelOutput` Nr0)rrJrr)last_hidden_staterH attentions)r9rJr)r$r%r'r ) rErr)rJrrencoder_statesall_attentions encoder_layer layer_outputsr0r0r1r\s        zPatchTSTEncoder.forwardNN) r]r^r_r`rrar=r(rdrcr r\rfr0r0rFr1rsrzG Base class for model's outputs, with potential hidden states. ) custom_introc@seZdZUdZdZejdBed<dZe ejdBed<dZ e ejdBed<dZ ejdBed<dZ ejdBed<dZ ejdBed<dZejdBed <dS) PatchTSTModelOutputa> last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, patch_length)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, num_channels, height, width)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. mask (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches)`, *optional*): Bool masked tensor indicating which patches are masked loc (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Mean of the input data (batch_size, sequence_length, num_channels) over the sequence_length scale (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Std of the input data (batch_size, sequence_length, num_channels) over the sequence_length patch_input (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, patch_length)`): Patched input to the Transformer Nr*rHr+rlocscaler)r]r^r_r`r*r( FloatTensorrrHrer+rr3r4rr0r0r0r1r2s r2z4 Output type of [`PatchTSTForPretraining`]. c@beZdZUdZdZejdBed<dZejdBed<dZ e ejdBed<dZ e ejdBed<dS)PatchTSTForPretrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): MSE loss. prediction_output (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction outputs of the time series modeling heads. Nlossprediction_outputrHr+) r]r^r_r`r8r(r5rr9rHrer+r0r0r0r1r7 r7z3 Output type of [`PatchTSTForRegression`]. c@r6)PatchTSTForRegressionOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): MSE loss. regression_outputs (`torch.FloatTensor` of shape `(batch_size, num_targets)`): Regression outputs of the time series modeling heads. Nr8regression_outputsrHr+) r]r^r_r`r8r(r5rr<rHrer+r0r0r0r1r;)r:r;z3 Output type of [`PatchTSTForPrediction`]. c@seZdZUdZdZejdBed<dZejdBed<dZ e ejdBed<dZ e ejdBed<dZ ejdBed<dZ ejdBed<dS) PatchTSTForPredictionOutputa! loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): MSE loss. prediction_outputs (`torch.FloatTensor` of shape `(batch_size, prediction_length, -1)`): Prediction outputs of the time series modeling heads. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. loc: (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*) Mean of the input data (batch_size, sequence_length, num_channels) over the sequence_length scale: (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*) Std of the input data (batch_size, sequence_length, num_channels) over the sequence_length Nr8prediction_outputsrHr+r3r4)r]r^r_r`r8r(r5rr>rHrer+r3r4r0r0r0r1r==s r=z7 Output type of [`PatchTSTForClassification`]. c@r6)PatchTSTForClassificationOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, num_targets)`): Prediction scores of the PatchTST modeling head (scores before SoftMax). Nr8prediction_logitsrHr+) r]r^r_r`r8r(r5rr@rHrer+r0r0r0r1r?]s r?z Base class for time series model's predictions outputs that contains the sampled values from the chosen distribution. c@s$eZdZUdZdZejdBed<dS)SamplePatchTSTOutputz sequences (`torch.FloatTensor` of shape `(batch_size, num_samples, prediction_length, num_targets)`): Sampled values from the chosen distribution. N sequences)r]r^r_r`rBr(r5rr0r0r0r1rArs rAinputtargetrKcCs || S)zc Computes the negative log likelihood loss from input distribution with respect to target. )log_prob)rCrDr0r0r1nlls rF input_tensorweightscCsr|dur3t|dk||t|}tj|r|j|dn|dd}|r-|j|d|S||S|j|dS)aj Computes the weighted average of a given tensor across a given `dim`, masking values associated with weight zero, meaning instead of `nan * 0 = nan` you will get `0 * 0 = 0`. Args: input_tensor (`torch.FloatTensor`): Input tensor, of which the average must be computed. weights (`torch.FloatTensor`, *optional*): Weights tensor, of the same shape as `input_tensor`. dim (`int`, *optional*): The dim along which to average `input_tensor`. Returns: `torch.FloatTensor`: The tensor with values averaged along the specified `dim`. Nrr#?min)r(where zeros_likeclamprr)rGrHr$weighted_tensor sum_weightsr0r0r1weighted_averages "  rQc PeZdZdZdeffdd Zdejdejdeejejejffdd Z Z S) PatchTSTStdScalerz Standardize features by calculating the mean and scaling along the first dimension, and then normalizes it by subtracting from the mean and dividing by the standard deviation. r9csVtt|dr |jnd|_t|dr|jnd|_t|dr&|j|_dSd|_dS)N scaling_dimrkeepdimT minimum_scalegh㈵>)r<r=hasattrrTr$rUrVrorFr0r1r=s  zPatchTSTStdScaler.__init__dataobserved_indicatorrKcCsz|j|j|jd}|d}||j|j|jd|}|||dj|j|jd|}t||j}|||||fS)C Parameters: data (`torch.Tensor` of shape `(batch_size, sequence_length, num_input_channels)`): input for Batch norm calculation observed_indicator (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Calculating the scale on the observed indicator. Returns: tuple of `torch.Tensor` of shapes (`(batch_size, sequence_length, num_input_channels)`,`(batch_size, 1, num_input_channels)`, `(batch_size, 1, num_input_channels)`) rUrIr")rr$rU clamp_minr(sqrtrV)rErXrY denominatorr3variancer4r0r0r1r\s  "zPatchTSTStdScaler.forward r]r^r_r`rr=r(rdrer\rfr0r0rFr1rSsrSc rR) PatchTSTMeanScalerz Computes a scaling factor as the weighted average absolute value along the first dimension, and scales the data accordingly. r9csltt|dr |jnd|_t|dr|jnd|_t|dr#|jnd|_t|dr1|j|_dSd|_dS)NrTrrUTrV绽|= default_scale)r<r=rWrTr$rUrVrcrorFr0r1r=s  zPatchTSTMeanScaler.__init__rXrYrKc Cs||j|jdd}|j|jdd}|tj|dd}|jdur:|jdd}tj|ddd}t||}n|jt|}t|dk||}tj||j d}||} |j sa|j|jd}| t ||fS)rZTr[rrJNrr#) absrr$r(rNrcsqueeze ones_likerLrVrUrM) rErXrYts_sum num_observedr4 batch_sumbatch_observationsrc scaled_datar0r0r1r\s  zPatchTSTMeanScaler.forwardr`r0r0rFr1rasrac sXeZdZdZdeffdd Z d dejdejdBdeejejejffd d Z Z S) PatchTSTNOPScalerz| Assigns a scaling factor equal to 1 along the first dimension, and therefore applies no scaling to the input data. r9cs@tt|dr |jnd|_t|dr|j|_dSd|_dS)NrTrrUT)r<r=rWrTr$rUrorFr0r1r=s  zPatchTSTNOPScaler.__init__NrXrYrKcCsBtj|ddj|j|jd}tj|ddj|j|jd}|||fS)a Parameters: data (`torch.Tensor` of shape `(batch_size, sequence_length, num_input_channels)`): input for Batch norm calculation Returns: tuple of `torch.Tensor` of shapes (`(batch_size, sequence_length, num_input_channels)`,`(batch_size, 1, num_input_channels)`, `(batch_size, 1, num_input_channels)`) Fr)r$rU)r(rfrr$rUrM)rErXrYr4r3r0r0r1r\ s  zPatchTSTNOPScaler.forwardrr`r0r0rFr1rlsrlc sLeZdZdeffdd ZdejdejdeejejejffddZZ S) PatchTSTScalerr9csRt|jdks|jdurt||_dS|jdkr"t||_dSt||_dS)NrTr)r<r=rrascalerrSrlrorFr0r1r=s  zPatchTSTScaler.__init__rXrYrKcCs|||\}}}|||fS)a> Parameters: data (`torch.Tensor` of shape `(batch_size, sequence_length, num_input_channels)`): Input for scaler calculation observed_indicator (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Calculating the scale on the observed indicator. Returns: tuple of `torch.Tensor` of shapes (`(batch_size, sequence_length, num_input_channels)`,`(batch_size, 1, num_input_channels)`, `(batch_size, 1, um_input_channels)`) )rn)rErXrYr3r4r0r0r1r\$s zPatchTSTScaler.forward) r]r^r_rr=r(rdrer\rfr0r0rFr1rms rmcsreZdZdeffdd Z     ddejdejdBdejdBdedBd edBd edBd ee Bfd d Z Z S) PatchTSTModelr9csft|t||_t||_|j|_|jj}|jr!t||_ nt |_ t ||d|_ |dS)N)r)r<r=rmrnr patchifier do_mask_inputrrmaskingrrrencoderr(rrFr0r1r=8s      zPatchTSTModel.__init__Nrpast_observed_mask future_valuesr)rJ return_dictrKc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur't|}|||\}} } ||} |jr@| | \} } n| | d} } |j | ||d}|sk|j |j |j f}|| | | | f}tdd|DSt|j |j |j | | | | dS)a Parameters: past_values (`torch.Tensor` of shape `(bs, sequence_length, num_input_channels)`, *required*): Input sequence to the model past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). future_values (`torch.BoolTensor` of shape `(batch_size, prediction_length, num_input_channels)`, *optional*): Future target values associated with the `past_values` output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers return_dict (`bool`, *optional*): Whether or not to return a `ModelOutput` instead of a plain tuple. Returns: `PatchTSTModelOutput` or tuple of `torch.Tensor` (if `return_dict`=False or `config.return_dict`=False) Examples: ```python >>> from huggingface_hub import hf_hub_download >>> import torch >>> from transformers import PatchTSTModel >>> file = hf_hub_download( ... repo_id="hf-internal-testing/etth1-hourly-batch", filename="train-batch.pt", repo_type="dataset" ... ) >>> batch = torch.load(file) >>> model = PatchTSTModel.from_pretrained("namctin/patchtst_etth1_pretrain") >>> # during training, one provides both past and future values >>> outputs = model( ... past_values=batch["past_values"], ... future_values=batch["future_values"], ... ) >>> last_hidden_state = outputs.last_hidden_state ```N)rr)rJcss|] }|dur|VqdSrr0)rvr0r0r1 z(PatchTSTModel.forward..)r*rHr+rr3r4r)r9use_return_dictrJr)r(rfrnrprqrrrsr*rHr+rer2)rErrtrur)rJrvrscaled_past_valuesr3r4patched_values masked_valuesrencoder_outputrr0r0r1r\Js67  zPatchTSTModel.forwardNNNNN) r]r^r_rr=r(rdrcrer2r\rfr0r0rFr1ro6s, rocs<eZdZdZdeffdd ZdejdejfddZZ S) PatchTSTMaskPretrainHeadz- Pretraining head for mask modelling r9csHt|jdkrt|jnt|_t|j|j |_ |j |_ dSNr) r<r= head_dropoutrrrrr@rlrlinearrrorFr0r1r=s   z!PatchTSTMaskPretrainHead.__init__ embeddingrKcCs:|||}|jr|ddddddddf}|S)a Parameters: embedding (`torch.Tensor` of shape `(bs, num_channels, num_patches, d_model)` or `(bs, num_channels, num_patches+1, d_model)` if `cls_token` is set to True, *required*): Embedding from the model Returns: `torch.Tensor` of shape `(bs, num_channels, num_patches, d_model)` or `(bs, num_channels, num_patches+1, d_model)` if `cls_token` is set to True Nr)rrr)rErr0r0r1r\s  z PatchTSTMaskPretrainHead.forwardrrr0r0rFr1rsrz* The PatchTST for pretrain model. csfeZdZdeffdd Z    d dejdejdBdedBdedBd edBd ee Bf d d Z Z S)PatchTSTForPretrainingr9cs4t|d|_t|d|_t||_|dS)NTr")r<r=rqrorrheadr(rorFr0r1r=s    zPatchTSTForPretraining.__init__Nrrtr)rJrvrKcKs|dur|n|jj}|j||||dd}||j}tjdd} | ||j} | jdd|j  |j d} |j } |sU|f|d d } | durQ| f| } | S| } | St | || |j d S) a Parameters: past_values (`torch.Tensor` of shape `(bs, sequence_length, num_input_channels)`, *required*): Input sequence to the model past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers return_dict (`bool`, *optional*): Whether or not to return a `ModelOutput` instead of a plain tuple. Returns: `PatchTSTForPretrainingOutput` or tuple of `torch.Tensor` (if `return_dict`=False or `config.return_dict`=False) Examples: ```python >>> from huggingface_hub import hf_hub_download >>> import torch >>> from transformers import PatchTSTConfig, PatchTSTForPretraining >>> file = hf_hub_download( ... repo_id="hf-internal-testing/etth1-hourly-batch", filename="train-batch.pt", repo_type="dataset" ... ) >>> batch = torch.load(file) >>> # Config for random mask pretraining >>> config = PatchTSTConfig( ... num_input_channels=7, ... context_length=512, ... patch_length=12, ... stride=12, ... mask_type='random', ... random_mask_ratio=0.4, ... use_cls_token=True, ... ) >>> # Config for forecast mask pretraining >>> config = PatchTSTConfig( ... num_input_channels=7, ... context_length=512, ... patch_length=12, ... stride=12, ... mask_type='forecast', ... num_forecast_mask_patches=5, ... use_cls_token=True, ... ) >>> model = PatchTSTForPretraining(config) >>> # during training, one provides both past and future values >>> outputs = model(past_values=batch["past_values"]) >>> loss = outputs.loss >>> loss.backward() ```NTrrtr)rJrvnone reductionr r#rbr)r8r9rHr+)r9rzrrr*rMSELossrrrrrHr7r+)rErrtr)rJrvr model_outputx_hatr8loss_val masked_lossr,rr0r0r1r\s,F  $ zPatchTSTForPretraining.forward)NNNN) r]r^r_rr=r(rdrcrer7r\rfr0r0rFr1rs& rcr)PatchTSTClassificationHeadr9csdt|j|_|j|_tjdd|_|jdkrt|jnt |_ t |j |j |j|_dSNr start_dimr)r<r=r pooling_typerFlattenflattenrrrrr@rrl num_targetsrrorFr0r1r=:s  z#PatchTSTClassificationHead.__init__rcCs|jr|dddddddf}n"|jdkr|jdd}n|jdkr+|jddj}n td|jd||}|||}|S) a[ Parameters: embedding (`torch.Tensor` of shape `(bs, num_channels, num_patches, d_model)` or `(bs, num_channels, num_patches+1, d_model)` if `cls_token` is set to True, *required*): Embedding from the model Returns: `torch.Tensor` of shape `(bs, num_targets)` Nrrr"r#rpooling operator  is not implemented yet) rrrrvaluesr?rrrrErpooled_embeddingrqr0r0r1r\Bs    z"PatchTSTClassificationHead.forwardr r0r0rFr1r9srz0 The PatchTST for classification model. csteZdZdeffdd Ze     ddejdejdBdedBdedBd edBd edBd e e Bfd d Z Z S)PatchTSTForClassificationr9csBt||jrtdd|_t||_t||_| dS)N+Setting `do_mask_input` parameter to False.F) r<r=rqloggerwarningrorrrr(rorFr0r1r=ds     z"PatchTSTForClassification.__init__Nr target_valuesrtr)rJrvrKc Ks|dur|n|jj}|j||||dd}||j} d} |dur)t} | | |} |sC| f|dd} | dur?| f| } | S| } | St| | |j|j dS)ac past_values (`torch.Tensor` of shape `(bs, sequence_length, num_input_channels)`, *required*): Input sequence to the model target_values (`torch.Tensor`, *optional*): Labels associates with the `past_values` past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Examples: ```python >>> from transformers import PatchTSTConfig, PatchTSTForClassification >>> # classification task with two input channel2 and 3 classes >>> config = PatchTSTConfig( ... num_input_channels=2, ... num_targets=3, ... context_length=512, ... patch_length=12, ... stride=12, ... use_cls_token=True, ... ) >>> model = PatchTSTForClassification(config=config) >>> # during inference, one only provides past values >>> past_values = torch.randn(20, 512, 2) >>> outputs = model(past_values=past_values) >>> labels = outputs.prediction_logits ```NTrrr)r8r@rHr+) r9rzrrr*rCrossEntropyLossr?rHr+) rErrrtr)rJrvrry_hatrr8rr0r0r1r\rs2-  z!PatchTSTForClassification.forwardr) r]r^r_rr=rr(rdrcrer?r\rfr0r0rFr1r^s. rz, The PatchTST for regression Model. cs8eZdZd dedeffdd ZdejfddZZ S) PatchTSTPredictionHeadNr9rcsDt|j|_|j|_|j|_|j|_|js|jr|j}n|j|}|jsvt|_ t|_ t|_ t |jD]8}|j tjdd|durW|j t||jn |j |||j |jdkrnt|jntq;dStjdd|_|durt||j|_n|||_|jdkrt|jnt|_dS)a num_patches (`int`): The number of patches in the input sequence. distribution_output (`DistributionOutput`, *optional*): The distribution output layer for probabilistic forecasting. If None, a linear output layer is used. r"rNr)r<r=share_projectionrrrrlrr projectionsdropoutsflattensrrrr@prediction_lengthget_parameter_projectionrrrr projectionr)rEr9rdistribution_outputr>rrFr0r1r=s0      (  $zPatchTSTPredictionHead.__init__rcCs|jr|dddddddf}n|jdkr|jdd}n|jdkr+|jddj}n|}|jseg}t|jD]%}|j||dd|ddf}|j ||}|j ||}| |q7t j |dd}n||}||}||}t|trtdd |D}|S|dd}|S) aj Parameters: embedding (`torch.Tensor` of shape `(bs, num_channels, num_patches, d_model)` or `(bs, num_channels, num_patches+1, d_model)` if `cls_token` is set to True, *required*): Embedding from the model Returns: `torch.Tensor` of shape `(bs, forecast_len, num_channels)` Nrrr"r#rrcss|] }|ddVqdS)r"rN)r*)rzr0r0r1rxryz1PatchTSTPredictionHead.forward..)rrrrrrrrrrrrr(r rrrrrer*)rErrrqrr0r0r1r\s.          zPatchTSTPredictionHead.forwardr) r]r^r_rrar=r(rdr\rfr0r0rFr1rs+rz, The PatchTST for prediction model. cseZdZdeffdd Z     ddejdejdBdejdBdedBd edBd edBd ee Bfd d Z e ddejdejdBd e fddZ ZS)PatchTSTForPredictionr9cst||jrtdd|_t||_|jdkrd|_n/|jdkr,t |j d|_n"|jdkr9t |j d|_n|jdkrFt |j d|_nt d|jt||jjj|jd |_|dS) NrFmse student_tr#normalnegative_binomialUnknown distribution output )r)r<r=rqrrrorr8rrrrrr?rrprrr(rorFr0r1r=&s$        zPatchTSTForPrediction.__init__Nrrtrur)rJrvrKcKs|dur|n|jj}|j||||dd}||j} d} |jr"| } n| |j|j} |durQ|jrF|jj| |j|jd} t | |} t | } n t j dd} | | |} |j}|j}|sq| f|dd}| durm| f|}|S|}|St | | |j|j||d S) aV Parameters: past_values (`torch.Tensor` of shape `(bs, sequence_length, num_input_channels)`, *required*): Input sequence to the model past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). future_values (`torch.Tensor` of shape `(bs, forecast_len, num_input_channels)`, *optional*): Future target values associated with the `past_values` output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers return_dict (`bool`, *optional*): Whether or not to return a `ModelOutput` instead of a plain tuple. Returns: `PatchTSTForPredictionOutput` or tuple of `torch.Tensor` (if `return_dict`=False or `config.return_dict`=False) Examples: ```python >>> from huggingface_hub import hf_hub_download >>> import torch >>> from transformers import PatchTSTConfig, PatchTSTForPrediction >>> file = hf_hub_download( ... repo_id="hf-internal-testing/etth1-hourly-batch", filename="train-batch.pt", repo_type="dataset" ... ) >>> batch = torch.load(file) >>> # Prediction task with 7 input channels and prediction length is 96 >>> model = PatchTSTForPrediction.from_pretrained("namctin/patchtst_etth1_forecast") >>> # during training, one provides both past and future values >>> outputs = model( ... past_values=batch["past_values"], ... future_values=batch["future_values"], ... ) >>> loss = outputs.loss >>> loss.backward() >>> # during inference, one only provides past values, the model outputs future values >>> outputs = model(past_values=batch["past_values"]) >>> prediction_outputs = outputs.prediction_outputs ```NTrr3r4rrrr )r8r>rHr+r3r4)r9rzrrr*rr4r3 distributionrFrQrrr=rHr+)rErrtrur)rJrvrrrr y_hat_outrr8r3r4rr0r0r1r\CsL>      zPatchTSTForPrediction.forwardcsr|jj}||d|dd}|jr.|jj|j|j|jdfddt|D}tj |dd}n|j d}t |d S) a  Generate sequences of sample predictions from a model with a probability distribution head. Parameters: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the future. past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Return: [`SamplePatchTSTOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, prediction_length, 1)` or `(batch_size, number of samples, prediction_length, num_input_channels)` for multivariate predictions. NF)rrurtr)rcg|]}qSr0samplerrr0r1rr#z2PatchTSTForPrediction.generate..rr#rB) r9num_parallel_samplesrrr>r3r4rr(r rrArErrtrrsamplesr0rr1generates   zPatchTSTForPrediction.generaterr)r]r^r_rr=r(rdrcrer=r\rrArrfr0r0rFr1r s>  nrcs8eZdZdZd deffdd ZdejfddZZ S) PatchTSTRegressionHeadz Regression head Nr9cst|j|_|j|_|j|_||_|j|j}t j dd|_ |j dkr,t |j nt |_|dur?t ||j|_dS|||_dSr)r<r= output_rangey_rangerrrrrlrrrrrrrr@rrr)rEr9rr>rFr0r1r=s   zPatchTSTRegressionHead.__init__rcCs|jr|dddddddf}n"|jdkr|jdd}n|jdkr+|jddj}n td|jd|||}||}|j du|j du@r_t ||j d |j d|j d}|S) aY Parameters: embedding (`torch.Tensor` of shape `(bs, num_channels, num_patches, d_model)` or `(bs, num_channels, num_patches+1, d_model)` if `cls_token` is set to True, *required*): Embedding from the model Returns: `torch.Tensor` of shape `(bs, output_dim)` Nrrr"r#rrrr) rrrrrr?rrrrrr(sigmoidrr0r0r1r\s    (zPatchTSTRegressionHead.forwardrrrr0r0rFr1rsrz, The PatchTST for regression model. cseZdZdeffdd Ze     ddejdejdBdejdBdedBd edBd edBd e e Bfd d Z e ddejdejdBd e fddZZS)PatchTSTForRegressionr9cst||jrtdd|_t||_|jdkrd|_n/|jdkr,t |j d|_n"|jdkr9t |j d|_n|jdkrFt |j d|_nt d|jt||j|_|dS) NrFrrr#rrr)r<r=rqrrrorr8rrrrrr?rrr(rorFr0r1r=s        zPatchTSTForRegression.__init__Nrrrtr)rJrvrKc  s|dur|njj}j||||dd}|j} d} |durIjr>j| } tfdd| D} t| |} t | } n t j dd} | | |} |sc| f|dd } | dur_| f| } | S| } | St | | |j |jd S) a# past_values (`torch.Tensor` of shape `(bs, sequence_length, num_input_channels)`, *required*): Input sequence to the model target_values (`torch.Tensor` of shape `(bs, num_input_channels)`): Target values associates with the `past_values` past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Whether or not to return a `ModelOutput` instead of a plain tuple. Examples: ```python >>> from transformers import PatchTSTConfig, PatchTSTForRegression >>> # Regression task with 6 input channels and regress 2 targets >>> model = PatchTSTForRegression.from_pretrained("namctin/patchtst_etth1_regression") >>> # during inference, one only provides past values, the model outputs future values >>> past_values = torch.randn(20, 512, 6) >>> outputs = model(past_values=past_values) >>> regression_outputs = outputs.regression_outputs ```NTrc3s |] }|djjVqdS)r N)rMr9r)ritemrEr0r1rxpsz0PatchTSTForRegression.forward..rrrr)r8r<rHr+)r9rzrrr*rrrerFrQrrr;rHr+) rErrrtr)rJrvrrrr8rrr0rr1r\9s<&      zPatchTSTForRegression.forwardcsb|jj}||d|dd}|j|jfddt|D}tj|ddd||jj }t |d S) a Generate sequences of sample predictions from a model with a probability distribution head. Parameters: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the future. past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Return: [`SamplePatchTSTOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, num_targets)`. NF)rrrtr)crr0rrrr0r1rr#z2PatchTSTForRegression.generate..rr#r r) r9rrrr<rr(r rMrrArr0rr1rs zPatchTSTForRegression.generaterr)r]r^r_rr=rr(rdrcrer;r\rrArrfr0r0rFr1rs@ Jr)rorrrrr)Nr)NFrrr0)Pr`rcollections.abcr dataclassesrr(rrr activationsrintegrations.deepspeedrmodeling_flash_attention_utilsr modeling_outputsr modeling_utilsr r processing_utilsr time_series_utilsrrrutilsrrrrconfiguration_patchtstr get_loggerr]rrrdrbr2r3rhlistrcrarrrrrrrrrr2r7r;r=r?rA distributions DistributionrFrQrSrarlrmrorrrrrrrr__all__r0r0r0r1s              V = D0< 3$8? "$7qp%X`>7