Building QERS in Public #2: The solution the project builds
How to answer a question that no single organization can answer
A pack of ten pouches of skipjack tuna in water sits on the shelf of a warehouse club. Ask where the tuna came from: which fishing trips and vessels, and whether, on any of those trips, species were caught that the international body managing tuna fishing in the eastern Pacific, the Inter-American Tropical Tuna Commission (IATTC), requires to be released, and were kept on board. The answer exists, but it is scattered: across the supplier, the third parties it contracts, the public bodies that regulate or monitor the chain and the retailer itself, and, inside each of them, across teams that name and describe the same things differently. And no organization will hand over its systems to another.
The question, then, is not where the data is. It is how to query the systems of the supplier, the third parties it contracts, the public bodies and the retailer with a single question, and get one answer, while each of them keeps its data in its own systems and under its own control. What follows are the decisions behind the architecture I am designing for that, each with its reason and with what I discarded.
Decision 1: leave the data where it lives
The first temptation is to centralize: copy the data of every organization into a single central database, run by one of them or by a third party, and query it there. I discarded it for a reason no technology removes: it requires every organization to hand over its data, and none will. The architecture is therefore decentralized: no organization hands its data or its systems to another, and each one keeps, governs and answers for its own.
That leaves the matter of how a question gets answered at all. The answer is federated: the question is answered by asking each owner for its part at the moment it is asked, and joining the parts, instead of first copying everything into one place. Each organization answers for its part, from its own systems, when someone asks.
Decision 2: do not ask for one model
Inside a single organization, federating would be enough, because the organization can agree on a single description of what its data means: which kinds of things it deals with, their properties and how they relate, an ontology. Across organizations, no such agreement can be required. The supplier will not adopt the retailer’s ontology, nor the retailer the supplier’s, and a public body will adopt neither.
So I discarded a second option: imposing one shared ontology on the whole ecosystem. Each organization keeps its own. What gets recorded instead is how they correspond: which term of one organization’s ontology corresponds to which term of another’s. That is called alignment.
Decision 3: rest on standards that belong to no one
If no participant’s model can be imposed, the common ground has to belong to none of them. It is made of semantic standards: public rules, owned by no single company, for writing data so that its meaning and its identity travel with it and the systems of different organizations can read it. The project rests on two.
Get Diego Soto’s stories in your inbox
Join Medium for free to get updates from this writer.
The first, RDF, writes data as simple statements of three parts: a thing, a relation and another thing. “This lot comes from this trip” is such a statement. The second, DPROD, describes what each organization publishes as a data product: what is offered, who owns it, how to reach it and under what conditions. One says what the data means; the other says what each organization makes available, and to whom.
What that changes for the tuna
Suppose the supplier records a fishing trip under one code and a public body records the same trip under another. Nothing in either record says that both describe the same trip. Once it is recorded that the two codes name the same trip, the question can follow it from one organization’s records to the other’s. That is shared identity: knowing that two identifiers, in different organizations, name the same thing.
Now suppose that, for one of them, a “trip” is a whole voyage, and for the other a single day at sea. Here the trouble is not the code but the word. Shared meaning is knowing that a term means the same for everyone who uses it, or exactly how the meanings differ. With both in place, data from different sources can be joined by what it means and by what it names, not by how it is stored. That is semantic data integration, and it is the heart of the design.
Decision 4: tie every question to a task and a decision
No question is asked for its own sake. Each one is tied to a task in a business process and to the decision it informs. The full tuna question shows why. Its last parts ask which stores received the lot, and which other lots of the same product come only from trips that carried a trained person whose job is to travel with the vessel and record what is caught, an on-board observer. Those parts are there because, if the supplier has not opened a corrective action, there is a decision to make about the lot.
And the one who asks need not be a person. Whoever carries out a task, whether a person, an AI agent or a machine, may ask, within the retailer or elsewhere in its ecosystem. In every case, the answer comes from data that each organization keeps under its own control.
Where it is tested
I am testing this design in my project, Towards Queryable Environmentally Responsible Sourcing (QERS), on Z-tow, a fictional warehouse club retailer, a company that runs warehouse clubs for paying members who buy in bulk. I built it, with its suppliers, its carriers and the public bodies that oversee them, from research in public sources, because no real retailer and its partners would open their systems for this. The pack of tuna is one of its products: item 980417, lot E2609–07.
What comes next
The next post on Medium is about the purpose of the project: what it sets out to do, what it will not do, and why it runs on a fictional retailer. In the meantime, if your organization’s systems need to share meaning and identity with those of others, I would like to talk.
About the scenario
Z-tow is a fictional warehouse club retailer, and the organizations of its ecosystem (its suppliers, carriers and any other company in it) are fictional too, invented to test this project. None of them is or represents a real company. Their organization, processes, systems, products and data are invented. The project never had access to the internal workings of any retailer, supplier, carrier or similar business: the scenario comes from public sources, from what any customer can observe, and from large language models. Some elements started from real products on public sale and from how real companies work as seen from outside, such as the kind of environmental claim on a product or the material of its packaging; names, brands and identifiers were replaced by invented ones, and no product of the scenario represents a real one. Lot numbers, item numbers and site codes are invented; any coincidence with a real identifier is accidental. Public bodies named in the scenario, such as the Inter-American Tropical Tuna Commission (IATTC), are real and their rules are public, but they appear in an invented case that implies no involvement or endorsement by them.
