John A. Zachman: Agile at 82 (85)

[This article was first published on InfoWorld in 2016. It is now reproduced here as this is the location where all my articles are supposed to be available. There have been small changes since the 2016 version. John Zachman is now 85 years old and still going strong.]

From June 13 until June 15 2016, I went to the Enterprise Architecture Conference Europe 2016. I was there to deliver the EA keynote (about Chess and the Art of Enterprise Architecture, narrated presentation here), but that also gave be the opportunity to go to presentations and workshops myself. One of these was by John Zachman, widely seen as the inventor of Enterprise Architecture and the author of the famous Zachman Framework®.

When I said I was going to visit John’s workshop (because it was a nice opportunity to actually hear from him in person), I was told by one of the other speakers: “Well, his ideas are not that relevant anymore these days.”

Now, I must admit that did not sound too strange to me. I had only superficially looked at his framework in the past. It had been presented to me a long time ago in a way that made me conclude it was a big hopeless waterfall that was totally unusable in practice and utopian in its reach. Maybe nice for designing and building commercial airliners or Channel tunnels, where designing is an initial stage that costs many millions, takes several years and produces a design of a product that has a life expectancy of several decades or more. Or nice in the deep past of enterprise architecture, when the world was still somewhat manageable with such approaches.

But I was wrong. If I took one thing from the workshop it was that Zachman’s framework is one thing above all else: misunderstood.

The Zachman ‘Framework’

So, for those who don’t know it, what is the Zachman Framework? It’s a way to divide everything you have to document about design in a certain way. It is — as John these days also explicitly tells people — an ontology. According to my dictionary, an ontology is “the branch of metaphysics dealing with the nature of being,” which doesn’t immediately sound like something practical. But, being a bit interested in philosophy, I know the shorthand for this, which is that an ontology describes ‘what there is’. In this case, John’s ontology describes what should be in a design and for that he has created a matrix that combines two divisions that are both as old as the way to Rome.

One axis, normally spread about the horizontal of the matrix, consists of the following fundamental elements of human analysis:

• What — Inventory
• How — Process
• Where — Distribution (Location)
• Who — Responsibility (Roles)
• When — Timing (Order of behavior)
• Why — Motivation

In the workshop, John stressed that he did not invent this and he does not take credit for it. He said he just re-uses what has been a basis of human analysis for as long as humans can remember. He noted that these categories have always been used for any subject of design, be it buildings or airplanes or anything else and that they thus also should apply to designing enterprises. His use is not a proposal for an EA mechanism, it is just saying that design is design, or in John’s words: “Architecture is architecture is architecture,” and thus that this generic ontology is by definition also applicable to enterprise architecture. It could not be not the case.

The second axis, normally positioned along the vertical, consists of several transitions from the more ‘abstract’ ideas to the more concrete descriptions (and finally implementation), which he said is as old as the fundamental insights of the Greek philosophers of 2500 years ago. These layers of intent are linked to certain stakeholders in his scheme:

• Executive — (Business) Context, or Identification
• Management — (Business) Definition
• Architect — System Logic, or Representation
• Engineer — Technology Physics, or Specification
• Technician — Configuration

Below this in the ontology is the actual instantiation of the design. John stressed this at the start of his workshop: the building is not architecture, it is an instantiation of an architecture (design). It is the transformation of design to something real.

John also explicitly stated a truism, namely that leaving out details in one area leaves it to interpretation when transforming one intention to the more concrete (“down” in the ontology) is just a matter of taking risks. He told me afterwards that “any facts about the enterprise that you do not make explicit are implicit, that is, you (and anyone or everyone) are making assumptions about those implicit facts. If the assumptions are correct, it is positive because you save time and money.

On the other hand, if the assumptions are incorrect, it is negative in that incorrect assumptions are the sources of defects which could range from inconvenient to cataclysmic. So, my Framework does not say you have to formalize, populate, make explicit any Enterprise facts at all. However, any facts that you do not formalize are potential sources of risk, risk of defects.” Note specifically the pragmatic nature of John’s approach to filling the squares. It is quite telling that John’s neutral description of what happens when you leave details out has been transformed by many of his disciples into a requirement to fill in all the details (which is impossible anyway).

Both axes have a different character. The horizontal axis is a division into (semi-) independent aspects of design. The vertical axis is a transformational axis, it has a waterfall somewhat built into it, where we go from top to bottom when going (transforming) ‘down’ from idea to actual implementations. According to John the waterfall is only that “any one Row must support the intent of the Row above it”.

John’s message is that both these axes are generic dimensions of any design activity. So, they must hold for designing enterprises too.

The surprises

Several issues surprised me during the workshop with John Zachman.

The first is that in 2016 John was 82, and he had a stamina and speed when trying to explain it all that was beyond many 40 year-olds. He’s a man with a mission, the mission being to let people understand his story. Incidentally, during that whole morning he lost his train of thought once, then told us “you’re watching a senior moment here,” after which he quickly returned to the story. Earlier that week, when I gave my own keynote, I think in the space of an hour I had at least three such senior moments, and I’m by far not 82.

The second surprise was that John explicitly said that — though the vertical axis is from more ideal to more concrete when going down — it is not about from less detail to more detail. Each of his squares can be detailed, or at least to the extent that details are necessary. Enterprise Architects often use the word “abstraction” to mean “less detail” because they use their abstraction as a way to make the complexity manageable for themselves and their audience, and John’s ontology has often been used in such a process. But John’s story is more subtle and the idea that going from abstract rough ideas to concrete details is not part of it.

The third surprise was that John in his workshop explicitly talked about going forward on the basis of incomplete descriptions. He explicitly stated that it is not the idea to go into 4-year super-waterfall and end up with the perfect design which is then implemented. No, John said that even in his favourite analogical world, that of designing and building large airplanes, it may happen that work is already progressing and that a change higher up may lead to scrap and rework. Now, as he explained it that way, it was easy to see the current world of Agile IT in front of you. Yes, you go ahead in your little part of the world, you do the best you can, but be prepared to do scrap and rework when needed. If you’re not prepared to do scrap and rework in agile, technological debt will build up.

And if there is too much scrap and rework, then you take a step back and say, “Wait, I need to halt my production and wait until certain fundamental questions have been answered”. Not because we need to wait until every design is done in excruciating detail, but because it becomes a pragmatic economic choice to do so.

So, I got in — thinking to get a presentation about the ultimate unusable waterfall, from high level abstractions to low level details– and I found out it’s use according to John himself has agile and pragmatism built in to the core.

The analysis

So, now having looked at it better and have it explained to me by its inventor, what do I think about John’s Enterprise Ontology?

Well, with respect to the horizontal axis, I think John re-uses a very pragmatic and usable shopping list in “What, How, Where, Who, When, Why”. It is important to make sure you have paid enough attention to each of these when creating designs/architectures.

For me, though, the ‘why’ is not part of the design and I have strong doubts about the feasibility of using modelling (hence structure) for the more abstract ‘why’. The doubts about modelling the ‘why’ are also related to philosophy, in this case for instance Wittgenstein. Take for instance all the ethical issues that may be part of the ‘why’. We have no ruleset for ethics, it is hardly ever strictly separable in good and bad (or in discrete and disjunct elements — which modelling requires), and so forth. So, if your task is designing a solution in the context of patient records, it is not doable to model the ‘why’ at all in enough detail such that you won’t get trouble when transforming to the more concrete. This, by the way, is part of what in my own keynote I called the cultural seduction, the assumptions that lead us architects to assume that certain approaches will work, while in reality they don’t.

But I do agree that the ‘why’ is important, so there is not much harm done when leaving it in. Besides, John’s ‘why’ is only abstract when in a vertical-sense we are still in the upper idea area of his ontology. Going down, and the ‘why’ might become very strict, concrete and doable requirements engineering.

The discussion on modelling the upper ‘why’ brings us also to the vertical axis: the route from ‘abstract’ ideas to concrete implementations. As mentioned before, John is neutral about the level of detail when going from idea to implementation and there I agree. There is for me a relation with the golden rule for outsourcing that I mention somewhere in my book. Sometimes, the requirement for a solution may be orders of magnitude less complex than the solution. There, outsourcing and waterfall work well, because sharply defining the requirements works well. If I want a secure data link with less than one in ten to the umpteenth bit errors, the requirement is small and exact, the actual implementation may be extremely complex, depending on the value of umpteen. However, I can also have a situation where the requirements are very complex, here and there fuzzy and even sometimes contradictory (think patient records or most strategic change initiatives) where the actual technical implementation may even be relatively simple, even if the requirements are complicated. Sometimes, going ‘down’ even means less complexity, not more.

So, I guess, outsourcings work well, and waterfall is efficient and effective, when ‘from abstract to concrete implementation’ coincides with ‘from abstract model (less detail) to more detailed description’. If this is not the case, then (fixed price) outsourcing is problematic and the waterfall is costly. But I digress, as usual.

Returning to the issue of requirements engineering, it might seem that when going from the abstract (ideas) to the concrete (implementation) it gets easier to be precise, but I doubt that too, mainly because the actual requirements at lower levels may be more complex, such as the fact that a choice made now in the technical domain may remain in effect for a long time, which means that the requirements for that choice cannot be as clear cut as the waterfall would suggest. Airplanes (John’s favourite analogy) such as Boeing 747s are complex, yes, but as an instrument they’re rather robust in their role, which is roughly the same as 40 years ago: carrying people and cargo safely and economically from A to B. IT systems in information-heavy enterprises have to contend with much larger variability in strategies and goals. That doesn’t mean that John is wrong about the importance of knowing what you want (designing it), just that it might be less feasible to be as detailed as one would like it to be.

Besides, I’m totally with John that being explicit as much as is feasible in a practical sense is a good idea. Just like Agile may just be a poor excuse to not do what you should do (design properly), or just like the enterprise architect’s shying away from detail into simplistic pictures can be a flight reaction, the problem of the complexity and unpredictability of our world doesn’t mean that we should completely forget about modelling and being as precise as we can (or need to be). We might not be able to do it as it is done in the airplane industry, but we certainly can often do a better job than we generally do. That is why I like modelling, e.g. in the enterprise architecture language ArchiMate (even if it is not perfect and not really open). And yes, that sometimes means we have to accept that there will be some ‘scrap and rework’ on our part. ‘Refactoring’ in modern agile-speak.

All in all, I’m not quite convinced that John’s layers and especially the top-to-bottom transformations (from intent to implementation) are always all that usable a distinction (which nicely coincides with the fact that these ancient Greek philosophers weren’t always right. Sometimes they were just plain silly as in “if justness is not the same as holy it must thus be unholy” — discrete logic followed to its silly extreme, see this short snippet about Protagoras). And we haven’t discussed all the subtle cause and effect dependencies that exist between the different squares in his ontology. E.g. if it turns out that human physical attendance is an important element of a certain part of the solution, then the impossibility to get those humans quickly from A (e.g. office) to B (e.g. off shore oil drill) in a technical sense might mean we have to design a completely different process high up, or use a different inventory high up, to get continuity. Hence, it is not just down from abstract idea to concrete implementation, sometimes issues of the concrete implementation have an effect on more abstract issues elsewhere in the ontology. It’s not a waterfall, it’s a tangled web (who remembers Literate Programming these days?).

Besides, there is the important and core issue of the unpredictability of the developments in all layers to take into account, though I do think some of that may fit in John’s scheme.

And finally, I’m not convinced that the stakeholder roles are as disjunct as John’s vertical axis suggests. E.g. it can be crucial for the work of the architect to be involved in the other layers, and sometimes maybe even be a bit of stakeholder him- or herself.

Hence, my experience tells me that John’s entire vertical distinction may not always be workable.

Conclusion

Even with its limitations, John’s Enterprise Ontology is a grand and useful idea. It is also often being misunderstood and misrepresented in two fundamental ways:

1. John’s “from (abstract) ideas to concrete implementation” has been interpreted as meaning “from less detail (another type of abstract) to more detail,” but no such assumption is actually part of his work. We are talking about different meanings of the word “abstraction” here (Uncle Ludwig’s “bewitchment by language” strikes again).
2. John’s setup has been interpreted as squares that need to be filled in completely (a huge waterfall) before you can implement anything, but John himself is more realistic, pragmatic, and agile about this and in any case the ontology does not tell you how to use it.

Maybe it is a good idea that John would take some of his amazing energy to actually combat those misrepresentations.

All in all, a very nice experience to meet with John and listen to him. In the playgrounds of the agile crowd (by the way, agile: another grand idea), and elsewhere, his reputation might not be that good, but frankly, I think people could do worse than listen to his story. Maybe it’s time some EA organization should create the “John A. Zachman Award for Enterprise Architecture”, if only to honour him and the important role he has played in getting our discipline on the road in the first place.

This year, 2019, I’ll be giving a half-day workshop of my own at EAC 2019:

And if you will not come to join my workshop, you can listen to John himself who is giving an EA keynote and a workshop.