5. Evaluation Approach

For evaluation purposes, a manually created ground truth (by Veruska Zamborlini) is provided as matching pairs between source and target, denoting which segment of the target has been transcribed in the source data. The target data (SAA) is composed of 4179 segments from 4 inventories while the source data (MDB) is of 200 segments that represent a partial transcription of the 4 inventories. Both data include segment duplicates discussed above as repetitions or ties.

In principle, for a given query-segment (Montias transcript), there exists only a single match in the target. However, because the target data is composed of segments from multiple inventories, segments with the exact same text can be found in the same inventory or in multiple inventories. Furthermore, a segment can also be split into multiple segments. This leads to finding more than one target segment for certain query-segment. This issue is addressed in the matching strategy.

5.1 The Evaluation Strategy

Broken Segments. When a segment is split into multiple segments (multiple lines in this context), the ground truth provides multiple segments as an answer, which highlights the existence of a wrong segment. This means that predicting any of the ground truth answers is correct. However, if multiple answers are given by the predictor for that query, then all predictions must be correct. For example, if the answer for a query-segment A is a set of wrongly broken segments {E, F, G}, predicting any element from the set is correct but predicting {A, F, C} is incorrect because no wrong element should be included in the prediction set.

Duplicated Queries. Consistency dictates that, for the same query asked several times, the same answer should be returned unless new information is available. For this reason, duplicates (segments with the exact same text) are removed from the query data. This brings the source query from 200 to 172. Like the broken segments, targets that have the exact same text should be returned as answers as a set. We denote this set as true ties.