Evaluate each ontology vendor on how much work its software performs and how much remains for your team or the vendor’s engineers. This includes generating the starter ontology, connecting it to physical data, reviewing proposed definitions, and maintaining mappings as sources change.
Atlan, the Context Layer for AI, takes an automation-first, human-governed approach. Context Agents mine a starter ontology from query history, column-level lineage, and BI activity, while domain owners review and certify the result.
| Field | Detail |
|---|---|
| What it is | A 30-question bank for evaluating ontology vendors on automation and the manual work left over |
| Lifecycle stages | Bootstrapping, mapping, change management, human review, serving, coverage, time to first value, hidden costs |
| Scoring scale | 0 unsupported claim, 1 documented evidence, 2 demonstrated on your data |
| What to record | Who performed the work and how many hours, your team separate from vendor services |
| POC window | Two weeks as a test window for one access-ready, scoped domain |
| Cost horizon | Year-one setup and year-two maintenance, priced separately |
Ontology vendor evaluation: the question behind every question
To help enterprises evaluate ontology vendors effectively, we have organized a question bank by lifecycle stage. Each table connects the question to ask with the evidence you should expect from a good answer and the response patterns that signal hidden manual work. Atlan should be held to every question in the bank, including Question 8.
Ask shortlisted vendors to demonstrate their answers on your data, then score each response using the same standard:
- 0: Unsupported claim. The vendor describes a capability without showing evidence.
- 1: Documented evidence. The vendor provides a relevant workflow, artifact, or measured result.
- 2: Demonstrated on your data. Your team can observe the workflow and verify the result.
Alongside each score, record who performed the work and how many hours they spent, separating your team’s time from vendor services. This shows whether a strong technical answer still leaves substantial manual effort with people.
Use what a metadata tooling RFP should include to structure the wider procurement process and the broader context layer evaluation criteria for requirements such as permissions and portability. If participants need a common baseline, first align on the ontology in AI and its core components.
Bootstrapping: does the vendor generate a starter ontology, or do your SMEs hand-model it?
A vendor should generate a first-pass ontology from evidence already in your data estate before asking SMEs to start from scratch. Hand-modeling can produce a strong ontology, but it makes rollout speed and accuracy dependent on recurring expert hours.
A credible context bootstrapping process reads query logs, lineage, BI logic, and glossary terms. Preserve the week-one output, then record the candidates generated, the share accepted, and the hours spent preparing inputs and correcting the draft.
According to Fathallah et al. (2025, revised 2026), relational schemas can be converted into ontologies with minimal human intervention, in experiments across three databases spanning two domains. Lippolis et al. (2025) found that generated ontologies can still contain common mistakes and vary in quality. Automated generation is real, but its output still requires expert review.
Use these questions to distinguish genuine bootstrapping from manual modeling presented as automation:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 1. Starting from our query logs, lineage, BI models, and glossary, what does your system generate in week one before any SME workshop? | Shows draft entities, relationships, and metric definitions from your sample, with source signals and confidence scores | Begins with SME workshops with nothing drafted, or shows a generic industry template |
| 2. What share of the starter ontology was drafted by software rather than typed by a person? | Provides a measured percentage from a comparable deployment or your POC, counted separately for concepts, relationships, and bindings, with preparation hours | “It depends,” without supporting data |
| 3. Can we inspect the evidence behind each generated concept and relationship? | Connects each proposal to relevant SQL, joins, lineage, BI logic, or glossary terms | Presents a model without showing where its contents came from |
| 4. How does the system handle missing, weak, or contradictory signals? | Flags uncertainty, separates proposed concepts from approved ones, and routes exceptions to the appropriate domain owner | Silently fills gaps or sends every decision to an SME workshop |
Mapping to physical data: Who writes the bindings?
Bootstrapping creates a first draft. Mapping determines which tables, columns, joins, and metrics supply the data behind each concept.
The ontology design process covers how concepts and relationships are defined. In a vendor evaluation, focus on the next step: how those definitions connect to physical data and who performs the work.
A credible vendor should infer candidate bindings from automated SQL lineage rather than manual mapping and observed SQL, show the supporting data lineage for AI, and route ambiguous cases to domain owners. Record connector setup, corrections, and hours because a long connector list alone does not prove automated mapping.
Test the mapping process on your own systems:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 5. How are ontology concepts bound to physical tables and columns, and who writes the mappings? | Candidate bindings inferred from lineage and SQL joins, shown with evidence, and proposed for approval; mappings are versioned and tested, and exceptions and manual hours are counted | Mappings are written by hand with no evidence trail, no tests, and no record of the hours spent |
| 6. How many systems can you map across in the POC: warehouse, lakehouse, BI, and SaaS applications? | Several named systems are connected through existing connectors, with setup requirements and coverage shown per source | Other sources deferred to an unscoped, unpriced phase two |
| 7. When two sources implement the same concept differently, how is the correct binding chosen? | Competing evidence is shown; a domain owner approves the binding, and the decision is recorded | The most-used logic becomes authoritative without review |
| 8. If you need formal reasoning or standards exchange: which OWL 2 profiles and RDF serializations can you import and export, and which semantic checks do you support? | Profiles and formats are named; a round trip preserves the ontology; required entailment and consistency checks are demonstrated | “Standards-compliant,” without a round-trip or reasoning test |
The W3C OWL 2 standard separates the exchange of ontology files from checking their meaning. If formal reasoning matters, ask the vendor to demonstrate the required inferences and consistency checks in the POC.
Change management: what happens when a schema changes?
Change management exposes hidden labor. Ask vendors to rename a column, then change a metric definition without changing its schema. A maintained active ontology should identify affected concepts, propose repairs, route them to an owner, and rerun tests.
According to Ananth Packkildurai, Data Engineering Weekly (2026):
“Version ontology changes. Test constraints and mappings. Review changes to definitions that affect downstream workflows.”
Schema and meaning changes both create context drift. Detecting context drift matters only when an alert leads to an owned, versioned, testable repair.
Also ask how the system tells a rename from a drop-and-add, and which changes it cannot see at all: a business rule can change while every schema and query stays the same, and only an owner will know.
Run both changes live and record interventions:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 9. Rename a source column. What happens next? | Lineage shows affected bindings; the repair is proposed, assigned, versioned, and tested | Your team files a ticket and waits for vendor engineers |
| 10. Change a metric definition without changing its schema. What detects it? | Changed SQL or BI logic, or a conflicting definition, triggers review | Only schema differences are monitored |
| 11. How do we see drift before an agent fails? | Alerts identify affected concepts, supporting evidence, and an owner; the vendor names the change types it cannot detect | Periodic manual reviews or silence until failure |
| 12. Who approves and tests the repair, and how long does it take? | A named owner approves it; tests rerun; the POC log records elapsed time and human hours | Manual edits with no regression test or time record |
Human review model: are experts on the loop or at the keyboard?
Experts should validate proposed definitions and resolve exceptions, rather than author every definition manually. Ask vendors to show the review interface, approval rules, correction workflow, and total reviewer effort.
In the OntoExtend study, Lippolis et al. (2026) asked six ontology engineers to evaluate generated fragments across two use cases. The engineers found that the fragments needed “minor to moderate revision before integration.” This supports using AI as a drafting assistant whose output experts review and refine, but it does not establish a universal revision rate across all ontologies.
Review should be part of a defined context development lifecycle, with approved corrections versioned and promoted into a shared context repository.
Use these questions to measure where expert judgment is applied and how much time it requires:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 13. When AI proposes a definition, who approves it, and in what interface? | A named steward reviews the proposal, supporting evidence, and confidence score in plain language; engineers who maintain bindings can still work in code | No accountable owner, or business experts must edit code or configuration to approve a definition |
| 14. What happens to high-confidence and low-confidence suggestions? | Items follow an approval policy you set; any auto-apply threshold is yours to configure, and the vendor shows how often items above it were later revised or rejected; low-confidence items enter a steward’s review queue | Everything publishes automatically, or every suggestion requires manual rewriting |
| 15. How do corrections from agents and users improve the shared ontology? | Corrections are captured, reviewed, versioned, and promoted into shared context | Fixes remain in one agent’s prompt or a spreadsheet |
| 16. For 100 proposals, how many are accepted unchanged, revised, rejected, or unresolved, and how many reviewer hours are required? | Reports every outcome against the same denominator and includes time spent on rejected items and the review backlog | Quotes an acceptance rate while excluding difficult or unresolved items |
Serving to agents: MCP, APIs, and more than one model
In an ontology-first AI architecture, the ontology should serve the same approved context to different agents without being rebuilt for each model. The MCP specification (current revision 2026-07-28) defines an open protocol for connecting LLM applications to external data and tools. It standardizes access, but does not verify that the supplied context is correct.
Ask vendors to demonstrate how MCP delivers business context and whether their model-agnostic context layer works beyond one agent stack:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 17. Show two different agent stacks consuming the same ontology. | Both retrieve the same version of the approved definitions in a live test, with permissions enforced per agent, and an approved change reaches both | Only the vendor’s own assistant works |
| 18. Which MCP, API, or SQL interfaces are available now? | Working calls, access controls, documentation, and deployment requirements are demonstrated | Interfaces appear only on a roadmap |
| 19. What changes when we add another model or agent framework? | The context remains reusable; required integration steps and hours are stated | Each consumer needs a separate export, prompt, or ontology copy |
Coverage: your whole estate or one platform?
Coverage is meaningful only against an agreed scope. Before the POC, inventory the systems, concepts, definitions, and bindings in the selected domain. Then measure what the ontology covers and name what remains outside it.
Extending an ontology beyond one warehouse requires more than a connector list. Distinguish ontology coverage from semantic-layer coverage, then test whether neutral context crosses the boundaries of individual platforms:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 20. What fraction of our agreed POC scope is covered? | Measures concepts and physical bindings across named systems against the scoped inventory | Quotes a connector count without a denominator |
| 21. Which systems, concepts, and definitions remain outside? | Provides a coverage map, names the gaps, and shows how they affect agent answers | Leaves exclusions unclear or covers only the easiest system |
| 22. Who extends coverage to another system or domain, and how much work is required? | Identifies connector, modeling, and certification work, with customer and vendor hours | Restarts discovery and manual modeling for every expansion |
Time to first value: what should a two-week POC prove?
Treat two weeks as a proposed test window for an access-ready, scoped domain, not a promise of enterprise-wide implementation. The POC should show measurable improvement on your questions and demonstrate what happens after a source or definition changes.
Use the time-to-value framework to identify prerequisites. Apply context testing to the information retrieved by the agent and combine context simulation with expert review:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 23. What can the agent answer correctly at the end of the POC that it got wrong at the start? | Shows before-and-after results using your questions, the same model settings, and owner-approved answers | Presents a polished demonstration on sample data |
| 24. Which prerequisites must be completed before the two-week clock starts? | Lists prerequisites, owners, elapsed time, and customer and vendor hours | Excludes setup work from the reported timeline |
| 25. Can we change a source column or metric definition during the POC? | Demonstrates detection, repair, approval, and retesting | Defers maintenance testing until after purchase |
| 26. What are the pass criteria, and which tests remain unresolved? | Uses agreed thresholds and retains a complete failure log | Reports only selected successful answers |
Run the POC in five steps:
- Choose one domain and name its owner.
- Provide SQL, lineage, BI logic, glossary terms, and source access; record preparation hours.
- Freeze real questions and owner-approved answers, including ambiguous terms and exceptions.
- Run before-and-after evaluations using the same questions and model settings.
- Change a column or definition, rerun the tests, and total the authoring, review, repair, and services hours.
Hidden costs: whose effort appears in the contract?
License fees are only part of the cost. Count customer and vendor effort to build, review, repair, and operate the ontology.
Use build vs. buy vs. bundle TCO to surface all costs, then include internal labor in the context-layer ROI. Compare year-one setup with year-two maintenance.
Ask vendors to provide commercial answers alongside technical ones:
| Question to ask | What a good answer sounds like | Red-flag answer |
|---|---|---|
| 27. Which roles and hours are required on our side and yours in years one and two? | Itemizes both sides and separates initial setup from recurring maintenance | An undefined standing ontology team or vague quarterly staffing |
| 28. Are professional services or embedded engineers required, recommended, or optional? | States scope, hours, rates, and percentage of year-one spend | Open-ended services scoped after discovery |
| 29. What work and charges follow a schema or definition change? | Names included remapping, approval, and retesting work, with limits | Each change becomes an unpriced services project |
| 30. What happens when the implementation team leaves? | Provides a runbook, skills transfer, ownership, and support terms | Only the original engineers can maintain it |
Complete this worksheet for each vendor using hours, loaded internal rates, and contracted fees:
| Effort or cost | Year one | Year two |
|---|---|---|
| Domain-owner and SME review | ___ | ___ |
| Internal engineering, mapping, and repair | ___ | ___ |
| Vendor services or embedded engineering | ___ | ___ |
| License, infrastructure, and usage fees | ___ | ___ |
| Expected changes and support assumptions | ___ | ___ |
Calculate annual cost as vendor fees plus internal hours at loaded rates. Estimate year-two changes from your own history: count schema, dbt or BI model, and metric changes over the last 6 to 12 months. Multiply that rate by the effort you measured in the POC for each kind of change, and stage more than one kind so the figure is not a single sample.
Services may still be valuable when their scope, cost, skills transfer, and ongoing obligations are explicit.
How does Atlan answer these questions?
For bootstrapping and mapping, Context Agents mine query history, lineage, and BI activity into candidate context. Atlan’s specialist agents include Orion for ontology relationships, Lexis for glossary terms, Nexus for term-to-asset links, Sage for metric conflicts, and Scout for usage-based prioritization. Domain owners resolve ambiguity and certify the output.
For change management and coverage, the Enterprise Data Graph connects source dependencies through column-level lineage. Rename a column and change a metric definition in the POC, then verify what Atlan detects, which systems are covered, and who approves the repair. Lineage can expose impact, but it does not automatically establish business meaning.
For review and time-to-value, Context Engineering Studio provides domain-scoped Bounded Context Spaces and evaluations built from dashboards, reports, and production queries. Domain experts add missing business knowledge and approve context through the Build, Test, Review, Approve, Deploy, Learn lifecycle.
For serving, versioned Context Repos and the Context Lakehouse expose approved context through MCP, A2A, SQL, and APIs. An MCP server for your context connects that meaning to consumers. Test the serving implementation with two agent stacks using the same approved definitions.
Atlan productizes recurring maintenance through agents and review workflows, and complex engagements can include forward-deployed engineers, so ask for scope, hours, skills transfer, and recurring cost. Atlan serves its ontology as governed, queryable context through the interfaces above. Where a program needs formal OWL inference, score Question 8 on a dedicated ontology-engineering tool that runs alongside the context layer.
Documented workflows score level 1 at most on the scale above, because none of them is a measurement on your data. Ask Atlan for the same acceptance counts, reviewer hours, and services share you would ask any vendor for.
On coverage, Sridher Arumugham, Chief Data Analytics Officer at DigiKey, describes the scope DigiKey set:
“Atlan is our context operating system to cover every type of context in every system including our operational systems.”
That is one customer’s description of scope, not a measured coverage figure. Test it against your own inventory with Questions 20 to 22.
Run these questions against Atlan too, starting with bootstrapping on your own signals and a record of the hours people spend.
Real stories from real customers: Shared definitions in production
"Atlan captures Workday's shared language to be leveraged by AI via its MCP server. As part of Atlan's AI labs, we're co-building the semantic layer that AI needs."
Joe DosSantos, VP, Enterprise Data and Analytics, Workday
"Context is the differentiator. Atlan gave our teams the shared vocabulary and lineage to move from reactive data management to proactive AI enablement."
Kiran Panja, Managing Director, Cloud and Data Engineering, CME Group
Conclusion: price the next change
Before signing, request one final demonstration: change an upstream definition and watch who does the work. If the vendor cannot show the repair path and estimate its recurring cost, the year-two effort line remains unpriced.
FAQs about ontology vendor evaluation
1. What questions should I ask an ontology vendor before buying?
Ask how it creates the starter ontology, maps it to data, gets expert approval, handles changes, and maintains it. Request a live demonstration and estimates of each side’s roles and hours.
2. How do I tell if an ontology vendor’s “automation” is real?
Watch the software use your data and respond to a schema change. Count every manual task needed to prepare inputs, correct results, approve changes, and maintain the ontology.
3. How long should an ontology POC take, and what should it prove?
Two weeks can work for one scoped domain once access and prerequisites are ready; treat it as a test window, not a rollout plan. It should prove answer quality, change handling, coverage, and human effort. A full rollout takes longer.
4. Can an ontology be generated automatically from existing databases and query logs?
Software can draft an ontology from schemas, queries, and lineage. Experts must still resolve unclear meanings and approve the result.
5. Who maintains the ontology after go-live, and how many people does it take?
Domain owners approve business meaning, while technical teams handle exceptions. Staffing depends on coverage and change frequency, so ask for an estimate based on your POC.
6. What happens to the ontology when a schema changes?
The system should find affected mappings and propose repairs for review and testing. Ask the vendor to demonstrate the process and identify any manual work.
7. How much of an ontology project’s cost is professional services?
There is no standard percentage. Ask what services are included, their share of year-one spending, and the recurring cost of maintenance and internal labor.
Sources
Research and practitioner sources cited above:
- Lippolis et al.: Ontology Generation using Large Language Models (2025).
- Lippolis et al.: OntoExtend: A Framework for Requirement-driven and Scalable Ontology Extension with LLMs (2026).
- Fathallah et al.: Retrieval-Augmented Generation of Ontologies from Relational Databases (2025, revised 2026).
- Ananth Packkildurai: What an Ontology for AI Agents Actually Needs (2026).
- Model Context Protocol specification (2026-07-28).
- W3C: OWL 2 Web Ontology Language Conformance (2012).
