o 7ti@s$ddlZddZddZddZdS)NcCsN|dd}|dd}|dd}|dd}|d d}td d |}|S) Nz n'tzn'tz ))z( (z" "z "z (['.,])z\1)replaceresub)stringr N/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/paws-x/utils.pygeneral_detokenizes     r cCs|d|ddS)Nr)lower)textr r r lowercase_first_lettersrcsRgfdd}gkrt}td|dt|d|dd|S)Ncs|ddvr;|ddvr;t|d|d<t|d|d<|ddr1|ddd|d<t|d|d<|S||S)N sentence1N sentence2).,;)r stripendswithrappenddoc empty_docsr r _process_docs z.process_docs_paraphrases.._process_doczFound z empty documents out of the z total docs in the dataset: cSs|ddvo |ddvS)Nrrrr rr r r )s z*process_docs_paraphrases..)lenprintfiltermap)datasetrlen_empty_docsr rr process_docs_paraphrasess r')rr rr'r r r r s