A tree test of a machine-readable agricultural taxonomy revealed two opposite problems: people could reach the wrong answer directly, or the right answer only after getting lost. The most useful finding changed the researchers’ model.
Five of six specialists selected a destination the knowledge model considered wrong. Separately, five of six took a direct path to their choice, without wandering through the hierarchy. The researchers looked at the choices, and changed the model.
That is the intriguing result of a 2025 IEEE Access study by Filipi Miranda Soares and colleagues. They applied a familiar UX method, tree testing, to an unfamiliar target: a machine-readable ontology of Brazilian agricultural products.
An ontology is more than a menu. It specifies concepts and their relationships in a form software can use. In this case, the Agricultural Product Types Ontology (APTO) helps standardize terms such as plants, animal products, oils, and processed commodities. The research team wanted to know whether specialists could navigate its classifications and whether the classifications made sense to them.
The striking result was not simply that the system was difficult to navigate. Some of the paths scored as failures exposed a gap in what its designers had considered a legitimate classification.
The tree that belonged somewhere else
One task asked participants where three commercially cultivated trees (araucaria, eucalyptus, and pine, Pinus) belonged. The model placed them under Useful plant, through the classes Wood-producing plant or Cellulose-producing plant. Both are defensible ways to describe their agricultural and industrial value.
Yet most participants headed toward Forest product. The researchers had used that category for products derived from trees, such as timber, rather than the trees themselves.
Only 17% selected one of the predefined correct paths. Yet 83% took a direct path to their chosen answer; the average task time was about 27 seconds. A person could be quite decisive and still be marked wrong by the test’s answer key. With six analyzed participants, those percentages describe individual choices in a small expert study, not a population estimate.
After reviewing the result, the authors accepted that trees cultivated for commercial forestry could themselves reasonably be considered forest products. They revised APTO so Wood-producing plant and Cellulose-producing plant could each belong under both Useful plant and Forest product. They did not have to erase the original classification to recognize another valid one.
A wrong destination can expose a mistake in the answer key, not merely a mistake by the person choosing it.
The opposite result was hiding in the same study
Another task asked participants to classify items such as cashew nuts, cocoa beans, and coconuts as edible nuts or oilseeds. Here the researchers recorded 100% task success: all six people eventually chose a valid path.
But only 33% followed a direct path. Others explored incorrect branches and backtracked; average completion time was about 42 seconds.
These two tasks tell very different stories when success is read alongside the route:
| Study task | Reached a predefined correct answer | Followed a direct path | What the pattern suggested |
|---|---|---|---|
| Commercial trees | 17% | 83% | An alternative expert classification deserved reconsideration |
| Nuts and oilseeds | 100% | 33% | A valid destination could still be hard to approach |
A dashboard showing only final success would miss that distinction. A dashboard showing only directness would miss it too. What matters is where specialists went, which relationships they expected to find, and whether those expectations should change the model.
But the user is not always right either
The paper contains an equally important counterexample. Some participants classified fruit pulp under Fruit. That is an understandable association, but the ontology needs to express precisely what the relationship means.
In this model, a class beneath another class typically means “is a kind of.” Fruit pulp is processed material derived from fruit; it is not simply another kind of raw fruit. So the researchers represented the relationship with derives_from, rather than making fruit pulp a subclass of Fruit.
This is the distinction that makes the study more useful than a call to follow majority preference. User behavior can uncover relationships the model omitted. Domain expertise is still needed to decide whether a relationship is a type of, part of, derived from, or simply a useful alternative route.
For a website, a second route might be a cross-link. For a machine-readable ontology, choosing the wrong relationship can alter what software is entitled to infer from the data. The two problems are connected, but their solutions are not interchangeable.
A more useful question for information architects
The study's contribution is a different way to inspect disagreement. Instead of automatically counting every unexpected path as user failure, a team can ask:
- What was the person trying to classify or find? The task needs to reflect a meaningful domain decision.
- Which relationship did their choice imply? Trace the route, not only the final answer.
- Was the model missing a legitimate relationship, or was a different type of relationship being confused with a parent category? Use domain review to decide what should change.
That is where tree testing contributes: it exposes the paths people take through a hierarchy, including direct failures and successful detours. Interviews or think-aloud sessions can then explore why they chose them; the paper's authors note that their own path data provided limited insight into participants' reasoning.
The broader implication reaches beyond website navigation. Teams building structured knowledge for search, data integration, and AI applications make classification decisions that may look tidy and technically valid while carrying unexamined assumptions. This study does not establish that tree testing improves chatbot answers or RAG accuracy. It demonstrates something earlier in the chain: evidence from specialists can change the relationships a machine-readable knowledge model encodes.
The researchers started with a hierarchy and an answer key. They finished with a better question about the domain itself: when knowledgeable people keep taking a route the model calls wrong, what does the model need to learn?
Research source
Soares, F. M., et al. (2025). “A Novel UX-Based Approach for Ontology Evaluation: Applying Tree Testing to the Agricultural Product Types Ontology.” IEEE Access, 13, pp. 137986 to 138004. Published paper (PDF). The study tested 11 tasks with six analyzed specialists (seven recruited, one excluded for task-time outlier); all reported percentages in this piece refer to the specific tasks cited.
