Exact match (EM) evaluates question-answering systems by measuring the percentage of predictions that exactly match at least one ground-truth answer. It is one of the main metrics used for SQuAD.

Saint Olaf College

Google

5 main methods: Accuracy and Error Rate; Precision / Recall / F1 score; Exact Match (EM); Mean Reciprocal Rank (MRR)

Text classification performance analysis

These are also primary metrics and are more often used than accuracy or error
rate for imbalanced test sets, e.g., the majority of the test samples have one class label. Precision and recall for binary classification are defined as below. The F1 score is the harmonic mean of the precision and recall, as in Eq. 3. An F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.

Precision / Recall / F1 score

MRR is often used to evaluate the performance of ranking algorithms in NLP
tasks such as query-document ranking and QA. MRR is defined below, where 𝑄 is a set of all possible answers, and 𝑟𝑎𝑛𝑘(𝑖) is the ranking position of the ground-truth answer.


Mean Reciprocal Rank (MRR)

Classification accuracy and error rate are primary metrics used to evaluate the performance of a classification model. Let TP, FP, TN, and FN denote the number of true positives, false positives, true negatives, and false negatives, respectively. The classification Accuracy is the proportion of true results (both true positives and true negatives) among the total number of cases examined: $$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$$ The Error Rate is the proportion of misclassified cases (both false positives and false negatives) among the total number of cases examined, which is also equal to 1 minus Accuracy: $$\text{Error Rate} = \frac{\text{FP} + \text{FN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} = 1 - \text{Accuracy}$$

Learn Before

Related