o i @sdddlmZdeededeeedffddZdeeded edeeeeeffd d Zd S) ) Tokenizerlist_valuereturn.cCs^|dg}t|ddD]\}}||kr&|gt|d|d|RS||q |fS)aJSplit a list of integers by a given value. Args: list_: The list to split. value: The value to split the list by. Returns: A tuple of lists of integers. Examples: >>> _split_integer_list_by_value([1, 2, 3, 4, 5], 3) ([1, 2], [3, 4, 5]) >>> _split_integer_list_by_value([1, 2, 3, 4, 5], 6) ([1, 2, 3, 4, 5],) >>> _split_integer_list_by_value([1, 2, 3, 4, 5], 1) ([1, 2, 3, 4, 5],) >>> _split_integer_list_by_value([1, 2, 3, 4, 5, 3, 5, 6, 7], 3) ([1, 2], [3, 4, 5], [3, 5, 6, 7]) rN) enumerate_split_integer_list_by_valueappend)rrresultiitemr U/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/experimental/utils.pyrs " r tokenizer control_tokencCs\||}t||^}}t|dkrtd|dt|dkr(td|d||dfS)zSplit a list of integers by a given control token. Raises: ValueError: If the control token is not found in the list or if it is found more than once. rzControl token z! not found in the list of tokens.rz, found more than once in the list of tokens.)get_special_tokenrlen ValueError)rrrcontrol_token_idfirstrestr r r-_split_tokens_by_one_occurrence_control_token s    rN)%mistral_common.tokens.tokenizers.baserlistinttuplerstrrr r r rs &