o ॵi@sLddlmZmZddlZddlZddlmZddlm Z Gddde Z dS))AnyDictN)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) OfaSudokuPreprocessorz+ OFA preprocessor for sudoku tasks c sLtt|j|||g|Ri||jjdd|_|jdd|_|jdd|_|jrg|_ g|_ t dD]-}t dD]&}|j |d|j |d|d krW|d ksc|j d |j d q=q7t |j |_ t |j |_ t ||j}t |j |g} t |j |g} t |j| |jg|_ t |j| |jg|_ d Sd S) zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) promptz solve the sudoku . seg_embeddingFmax_struct_length rrN)superr__init__cfgmodelgetinstruction_textr r input_puzzle_rowinput_puzzle_colrangeappendtorchtensor zeros_like tokenize_textcatbos_itemeos_item) selfr model_dirmodeargskwargsidxjdx instruct_segrr __class__W/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/preprocessors/ofa/sudoku.pyrsH          zOfaSudokuPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)r!rTRAIN_build_train_sample_build_infer_sample)rr+r)r)r*__call__:s   zOfaSudokuPreprocessor.__call__cCsl||}|d}|}d|d|j}|j|dd|d<t|j |dddg|d<|S) z build sample for training tasks. step 1. execute the `_build_infer_sample` function to get a batch sample for inference. step 2. process the label data for training. label NF)add_bostargetprev_output_tokens) r/lowerstripsplitjoinmax_tgt_lengthrrrr)rr+sampler4target_token_listr)r)r*r.@s z)OfaSudokuPreprocessor._build_train_samplecCsd|jvr d|vs Jd||jd}d|d|j}|||j}|d|j|j}d|d}|j rH|j |d<|j |d<d |jvr`|jd |vr`d ||jd |d <|S) a- build sample for inference tasks. step 1. Get the input random masked sudoku text input, which shold be generated like below pseudo code. >>> sudo = np.random.randint(1, 9, size=(9, 9)) # a pseudo sudoku >>> sudo_text = " | ".join(" : ".join(str(c) for c in row) \ >>> for row in sudo) step 2. Limit the length, tokenize the input text and add the bos token to the front of the input as source input. step 3. Add a pseodo ids for every input. textz;there must be `text` column in task key map and source datar2Ng)idsourceseg_row_tokensseg_col_tokenssolutionz {}r1) column_mapr:r7r8r9r rrmax_src_lengthr rrformat)rr+r>src_itemr<r)r)r*r/Qs$     z)OfaSudokuPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__r INFERENCErrstrrr0r.r/ __classcell__r)r)r'r*r s"*"*r) typingrrnumpynprmodelscope.utils.constantrbaserrr)r)r)r*s