o
5ti- @ s d dl mZmZmZmZmZmZmZmZ d dl m
Z
m Z m Z d dl
mZ ddlmZ ddlmZmZ ddlmZmZmZ dd lmZ erPd d
lmZm Z m Z m Z d Z G d d
d
eZ dS ) )
TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)
ClassLabelDatasetSequence)Literal )EvaluationModule)add_end_docstringsadd_start_docstrings )"EVALUATOR_COMPUTE_RETURN_DOCSTRING EVALUTOR_COMPUTE_START_DOCSTRING Evaluator)
DatasetColumn)PipelinePreTrainedModelPreTrainedTokenizerTFPreTrainedModela
The dataset input and label columns are expected to be formatted as a list of words and a list of labels respectively, following [conll2003 dataset](https://huggingface.co/datasets/conll2003). Datasets whose inputs are single strings, and labels are a list of offset are not supported.
Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("token-classification")
>>> data = load_dataset("conll2003", split="validation[:2]")
>>> results = task_evaluator.compute(
>>> model_or_pipeline="elastic/distilbert-base-uncased-finetuned-conll03-english",
>>> data=data,
>>> metric="seqeval",
>>> )
```
For example, the following dataset format is accepted by the evaluator:
```python
dataset = Dataset.from_dict(
mapping={
"tokens": [["New", "York", "is", "a", "city", "and", "Felix", "a", "person", "."]],
"ner_tags": [[1, 2, 0, 0, 0, 0, 3, 0, 0, 0]],
},
features=Features({
"tokens": Sequence(feature=Value(dtype="string")),
"ner_tags": Sequence(feature=ClassLabel(names=["O", "B-LOC", "I-LOC", "B-PER", "I-PER"])),
}),
)
```
For example, the following dataset format is **not** accepted by the evaluator:
```python
dataset = Dataset.from_dict(
mapping={
"tokens": [["New York is a city and Felix a person."]],
"starts": [[0, 23]],
"ends": [[7, 27]],
"ner_tags": [["LOC", "PER"]],
},
features=Features({
"tokens": Value(dtype="string"),
"starts": Sequence(feature=Value(dtype="int32")),
"ends": Sequence(feature=Value(dtype="int32")),
"ner_tags": Sequence(feature=Value(dtype="string")),
}),
)
```
c $ s e Zd ZdZdg iZd/ fdd Zdeee deee d efd
d Z dee d efd d
Z
de ee f deded ef fdd Z
d0de ededdf de d de d def fdd
Zeeeee ! "d1de ededdf de ee f d#ee d$ed%e eef dee ed&f d'ed( d)ed*edee d+ee deded ee d,eeeef ef f d-d.Z Z S )2 TokenClassificationEvaluatora(
Token classification evaluator.
This token classification evaluator can currently be loaded from [`evaluator`] using the default task name
`token-classification`.
Methods in this class assume a data format compatible with the [`~transformers.TokenClassificationPipeline`].
ignore_labelstoken-classificationNc s t j||d d S )N)default_metric_name)super__init__)selftaskr __class__ [/home/ubuntu/.local/lib/python3.10/site-packages/evaluate/evaluator/token_classification.pyr f s z%TokenClassificationEvaluator.__init__ predictionswordsjoin_byc C s g }t |D ]U\}}g }| || |}d} |D ]=}
|| d |
d k r2| d7 } || d |
d k s$|| d |
d krB|d q|| d |
d krU||| d q|| qd|iS )aW
Transform the pipeline predictions into a list of predicted labels of the same length as the true labels.
Args:
predictions (`List[List[Dict]]`):
List of pipeline predictions, where each token has been labeled.
words (`List[List[str]]`):
Original input data to the pipeline, used to build predicted labels of the same length.
join_by (`str`):
String to use to join two words. In English, it will typically be " ".
Returns:
`dict`: a dictionary holding the predictions
r startr Oentityr' ) enumeratewords_to_offsetsappend) r! r' r( r) predsi
predictionpred_processed
words_offsets token_index word_offsetr% r% r& predictions_processori s z2TokenClassificationEvaluator.predictions_processorc C sD g }d}|D ]}|t | d }|||f |t | d }q|S )a
Convert a list of words to a list of offsets, where word are joined by `join_by`.
Args:
words (`List[str]`):
List of words to get offsets from.
join_by (`str`):
String to insert between words.
Returns:
`List[Tuple[int, int]]`: List of the characters (start index, end index) for each of the words.
r r )lenr/ )r! r( r) offsetsr* wordendr% r% r& r. s
z-TokenClassificationEvaluator.words_to_offsetsdata input_column label_columnc
s t || t|j trt|j| ts tdt|j| jt}|rC|j| jj}dd t |D fdd|| D }n|j| jj
drQt d|| }d|i}|
fd d
}t|} || fS ) NzYTokenClassificationEvaluator expects the input and label columns to be provided as lists.c S s i | ]\}}||qS r% r% ).0r1 labelr% r% r&
s z=TokenClassificationEvaluator.prepare_data..c s g | ] } fd d|D qS )c s g | ]} | qS r% r% )r? label_id id_to_labelr% r&
s zHTokenClassificationEvaluator.prepare_data...r% )r? label_idsrC r% r& rE s z=TokenClassificationEvaluator.prepare_data..intz[References provided as integers, but the reference column is not a Sequence of ClassLabels.
referencesc s | iS )N)join)x)r= r) r% r& s z;TokenClassificationEvaluator.prepare_data..)r prepare_data
isinstancefeaturesr
ValueErrorfeaturer
namesr- dtype
startswithNotImplementedErrormapr )
r! r< r= r> r) labels_are_int
label_listrH
metric_inputspipeline_inputsr# )rD r= r) r& rL s*
z)TokenClassificationEvaluator.prepare_datamodel_or_pipeliner r r tokenizer)PreTrainedTokenizerBaseFeatureExtractionMixinfeature_extractordevicec sF t ||||}|dgfi | j}|d d d d u r!td|S )Nz2003 New York Gregoryr r* zTokenClassificationEvaluator supports only pipelines giving 'start' index as a pipeline output (got None). Transformers pipelines with a slow tokenizer will raise this error.)r prepare_pipelinePIPELINE_KWARGSrO )r! rZ r[ r^ r_ pipe dummy_outputr# r% r& r` s z-TokenClassificationEvaluator.prepare_pipelinesimpleffffff?' tokensner_tags subsetsplitmetricr strategy)rd bootstrapconfidence_level n_resamples random_statereturnc C s i }| |
| | j|||d}| j|| |
|d\}}| j|||
d}| |}| ||\}}| ||| |}|| | j||||| | d}|| || |S )ac
input_column (`str`, defaults to `"tokens"`):
The name of the column containing the tokens feature in the dataset specified by `data`.
label_column (`str`, defaults to `"label"`):
The name of the column containing the labels in the dataset specified by `data`.
join_by (`str`, *optional*, defaults to `" "`):
This evaluator supports dataset whose input column is a list of words. This parameter specifies how to join
words to generate a string input. This is especially useful for languages that do not separate words by a space.
)r< rj rk )r< r= r> r) )rZ r[ r_ )rl rX rm ro rp rq ) "check_for_mismatch_in_device_setup load_datarL r` prepare_metric
call_pipeliner7 updatecompute_metric)r! rZ r< rj rk rl r[ rm ro rp r_ rq r= r> r) resultrX pipe_inputsrb r' perf_resultsmetric_resultsr% r% r& compute s,
z$TokenClassificationEvaluator.compute)r N)NNN)NNNNNNrd re rf NNrg rh ri ) __name__
__module__ __qualname____doc__ra r r r strr7 r. r r rL r rG r` r r r r TASK_DOCUMENTATIONr r r
floatr r r}
__classcell__r% r% r# r& r Z s &'&#
r N)!typingr r r r r r r r datasetsr
r r typing_extensionsr
moduler utils.file_utilsr r baser r r utilsr transformersr r r r r r r% r% r% r& s ( <