Evaluation
A workflow for evaluating mapping predictions against manually curated mappings.
The Ontology Alignment Evaluation Initiative (OAEI) has produced and evaluated benchmarks for ontology mapping software across biology, medicine, ecology, digital humanities, archaeology, and other disciplines since 2004. In order to submit, developers must ensure their ontology mapping software consumes ontologies in the Web Ontology Language (OWL) format and outputs mappings in the Expressive and Declarative Ontology Alignment Language (EDOAL) format, which can then be automatically evaluated by the OAEI’s Alignment API and Alignment Server. A table of past OAEI calls and results are available below.
During its two-decade runtime, the OAEI consistently reuses the same benchmarks. For example, the largebio task for mapping between the Foundational Model of Anatomy (FMA) ontology, Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT), and United States National Cancer Institute Thesaurus (NCIT) ran between 2011 and 2022 before being incorporated into the Bio-ML task, which still runs as of 2026.
This presents several opportunities to go beyond OAEI, in order to:
adopt a better semantic mapping format and software ecosystem
store and manually curate the results of mapping prediction
maintain old benchmarks and create new ones
retire benchmarks for which ontology alignment has been completed
The Simple Standard for Sharing Ontological Mappings (SSSOM) and its associated software ecosystem are still under active maintenance (whereas the alignment API project has not been updated since 2021), are already considerably better documented than the alignment API and EDOAL, and adopt much more straightforward languages (Python instead of Java) and formats (TSV instead of XML).
Community repositories for semantic mappings like Biomappings demonstrated how an open data, open
code, and open infrastructure (O3)
approach democratizes the storage and curation of semantic mappings. The Biomappings
project itself led to the development of the sssom_curator software to wrap
prediction pipelines and provide an interactive curation interface for end users.
The goal of the SSSOM-Pydantic evaluation pipeline is to build on existing tools for
extracting mappings from ontologies (e.g., pyobo), curated resources like
Biomappings, and easily reusable prediction workflows like SSSOM-Curator to
automatically construct new benchmarks based on existing SSSOM documents then
automatically calculate statistics about alignment completion (i.e., how many more
curations are needed to check all predicted mappings, and how many more curations are
needed to complete the alignment?) and the correctness of the prediction software (e.g.,
accuracy, precision, recall, \(F_1\)).
Until all predictions are curated, the accuracy, precision, recall, and \(F_1\) are an estimation of the true metrics, since the positive and negative manually curated mappings likely are not complete and therefore have some bias in which things were curated (e.g., I always curate the easiest first, leading towards a skew that more of my manual curations result in positive calls).
In the following example, three sources of mappings are combine for the evaluation:
Mappings from Medical Action Ontology (MAXO) extracted using
pyobo, which include mappings to Medical Subject Headings (MeSH) with no metadata, so they default tooboInOwl:hasDbXrefas a predicate andsemapv:UnspecifiedMappingas a justification.Manually curated mappings from Biomappings, which includes previously curated mappings between MAXO and MeSH with high precision predicates and justification.
Mappings predicted by the
sssom_curatorbetween MAXO and MeSH with lexical matching
$ pyobo lookup sssom maxo -o maxo.sssom.tsv
$ sssom_pydantic subset \
-i https://w3id.org/biopragmatics/biomappings/sssom/biomappings.sssom.tsv \
--prefix maxo \
--target-prefix mesh \
--no-exclude-negatives \
--no-exclude-unsure \
--exclude-predicted \
-o biomappings-maxo-mesh.sssom.tsv
$ mkdir maxo-mesh-predictions
$ sssom_curator init --directory maxo-mesh-predictions
$ sssom_curator -p maxo-mesh-predictions predict lexical mesh maxo
$ sssom_pydantic evaluate \
-i maxo.sssom.tsv \
-i biomappings-maxo-mesh.sssom.tsv \
-i maxo-mesh-predictions/data/predictions.sssom.tsv \
--accept-unspecified
This workflow pools arbitrary SSSOM files then stratifies them into positive, negative,
predicted (positive), and predicted negative mappings using the stratify()
function. When extending this workflow to several other OBO Foundry ontologies mapping
to MeSH, a table like this is produced:
Prefix 1 |
Prefix 2 |
Completion |
Accuracy |
Precision |
Recall |
\(F_1\) |
|---|---|---|---|---|---|---|
7.9% |
98.2% |
98.9% |
99.2% |
99.1% |
||
26.9% |
53.4% |
90.8% |
47.6% |
62.5% |
||
50.0% |
61.9% |
66.7% |
85.7% |
75.0% |
||
29.7% |
93.5% |
93.3% |
100.0% |
96.6% |
||
32.5% |
80.3% |
82.6% |
96.1% |
88.8% |
||
1.9% |
43.6% |
68.0% |
45.9% |
54.8% |
||
12.2% |
96.6% |
98.8% |
97.7% |
98.3% |
||
43.3% |
86.9% |
100.0% |
86.9% |
93.0% |
||
17.6% |
95.8% |
95.8% |
100.0% |
97.9% |
||
39.6% |
88.9% |
100.0% |
88.9% |
94.1% |
||
44.8% |
81.5% |
80.8% |
100.0% |
89.4% |
||
14.6% |
95.2% |
95.2% |
100.0% |
97.6% |
||
25.8% |
72.6% |
98.4% |
73.5% |
84.1% |
||
7.1% |
12.2% |
98.7% |
12.2% |
21.7% |
||
69.4% |
64.1% |
91.2% |
53.8% |
67.6% |
||
0.3% |
50.0% |
50.0% |
100.0% |
66.7% |
||
44.7% |
98.7% |
98.7% |
100.0% |
99.3% |
Note that lexical matching typically has a high precision (i.e., most predictions are right) but lower recall (i.e., some potential predictions are missed). Given the problem domain that (almost all) ontologies don’t have one-to-many or many-to-one mappings, then it’s also possible to identify entities for which there is no mapping between two given resources and further increase the accuracy of the accuracy metric.
Looking forward, this workflow can be arbitrarily extended with additional curations of mappings in Biomappings (or any other manually curated SSSOM files or ontologies) to be applied. It currently doesn’t automatically stratify based on mapping tool, but if there are additional ones put into use (such as LeonMap, implemented by Harshit Soni and suggested here), then this is an important step towards building an automated, rerunnable evaluation workflow.
OAEI Calls and Publications
Year |
Call |
Publication |
|---|---|---|
2026 |
||
2025 |
||
2024 |
||
2023 |
https://ora.ox.ac.uk/objects/uuid:e167c7dc-72cd-476a-ba23-d4bcc86e0b60 |
|
2022 |
||
2021 |
||
2020 |
||
2019 |
||
2018 |
||
2017 |
||
2016 |
||
2015 |
||
2014 |
||
2013 |
||
2012 |
||
2011 |
||
2010 |
||
2009 |
||
2008 |
||
2007 |
||
2006 |
||
2005 |
||
2004 |
Functions
|
Stratify and evaluate predicted mappings against curated mappings. |
|
Stratify mappings into a positive, negative, and predicted (positive) set. |
Classes
|
An evaluation tuple. |
|
A 4-tuple of mapping dictionaries. |