This post is a transcript of my keynote at RuleML/DecisionCamp on July 8, 2016.
I was surprised to be invited to speak here about DMN, because the inventors of DMN are here in the room but I am not one of them. While they were off inventing DMN, I was working in the related area of business process management, most recently focused on business process modeling using BPMN.
But that background is relevant to my talk today because DMN, the new decision modeling standard from the Object Management Group (OMG), is now attempting to replicate the success of BPMN. Whether DMN succeeds or fails in that attempt, I believe, depends on whether implementers heed the lessons of the BPMN experience. OMG measures the success of its standards by breadth of adoption, and by that measure BPMN has been very successful, widely adopted by both business users – notoriously averse to standards of any kind – and technical users. Acceptance by business was, from the start, always an explicit goal of BPMN, as it is now for DMN.
What I really meant to call this talk is “DMN as a Business-Friendly, Tool-Independent, and Fully Executable Decision Modeling Language.” Because these define DMN’s fundamental promise: First, business users, this language is for you; second, it works the same in any DMN tool; third, and most important: what you model is what you execute. That is a powerful promise, because it lets business people themselves define and maintain the decision logic that drives their critical operations, in contrast to traditional practice in which they are limited to writing requirements for programmers who have no direct knowledge of the business. And this promise is disruptive. If DMN is able to fulfill it, it will change both the practice of decision management and the competitive landscape of decision management software.
Fulfilling the promise requires implementation of all 3 of DMN’s key attributes – business-friendliness, tool-independence, and executability. And that in turn requires adoption of a standard expression language. DMN defines one, called FEEL, that is both business friendly and rich enough to handle real-world decision logic, along with a tabular decision logic format called boxed expressions. Surprisingly, however, DMN tools are not required to implement them. While FEEL and boxed expressions are essential to DMN’s promise, the struggle over their implementation continues, and is central to my story today.
So my talk has 3 parts. Part 1, The Promise, gives a little background on DMN and its promise, and what realizing that promise will entail. Part 2, How the language works, explains how DMN models defined entirely through diagrams and tables can be executable. And Part 3, Meeting the Implementation Challenge, discusses the difficulties that 3 months ago made it look unlikely that DMN would ever deliver on its promise, and ends with recent developments that suggest that DMN is not only likely to achieve its promise after all, but will begin to do so this year.
Three months ago, when I began thinking about this talk, it seemed unlikely that DMN would ever fulfill its promise, because even the objective of full business-empowerment was not widely shared by its key implementers, those active in the DMN task force in OMG.
Maybe, they said, DMN should simply provide a model-driven way to generate business requirements. Programmers would then translate those requirements into a tool-specific rule language for execution. That is much better than the current practice of text-based requirements documents, and it would make DMN implementation much easier. Others said, we can do more than that. Business should be able to model the decision logic, not just the requirements, and test it for completeness and consistency. But a business-friendly decision modeling language must be limited, they said, to simple things, like decision tables that select from an enumerated list, and simple arithmetic expressions. But that’s it. For anything more, you need to exit the standard and go back to writing requirements for programmers.
In fact the DMN spec itself, recognizing this diversity of opinion about the objectives, formally blesses the idea of partial implementation, defining 3 levels of DMN conformance. Conformance Level 3 embodies the full promise, the disruptive vision, including model-driven decision requirements, business-friendly decision logic definition, and support for FEEL, a tool-independent executable expression language. It provides all 3 legs of a strong stable platform…
Level 2 says, yeah, all of that… except the strong part. Because we don’t think business users are ready for FEEL. So we’ll just use a tiny subset of FEEL, called Simple FEEL or S-FEEL. It’s limited to decision tables that select from enumerated values and can do simple arithmetic. If your logic needs more, leave that to the programmers. Because we don’t think business users want to deal with that kind of logic. So Level 2 is also a stable platform, but a weak one. It won’t hold much weight. If you want to carry any weight, you will need to exit the standard language and let the programmers do it.
And Level 1 says, actually business users don’t want to even think about decision tables and other kinds of decision logic at all. They just need a better way, a model-driven way, to define business requirements for the programmers. Level 1 is business-friendly and tool-independent, but the logic is not precisely defined, so it cannot be tested. So this is like a 2-legged stool. Long-term, we know how that goes.
And there is also another level, call it Level 2a, which was not anticipated by the DMN spec, in which the tool provides advanced FEEL-like expressions – but uses the tool’s own language. So it’s business-friendly, executable, but not tool-independent. That’s actually another 2-legged stool, and not so different from what the commercial decision management suite vendors already offer.
I say fulfilling the promise seemed unlikely because, among those vendors active in the DMN task force, only one seemed to have any interest in implementing Level 3. It was just too hard, and besides, Level 1 and 2 already offered a big improvement over current decision management practice. It was not just that partial implementation was a faster way to get a tool to market. Vendors were arguing that generating requirements for implementation in some other engine-specific rule language was a viable long-term strategy. I found that consensus deeply disappointing, because the lesson of BPMN over the past decade is that in the long run, that approach doesn’t work. The promise that works is What You Model is What You Execute.
And then, suddenly, things started to change. In April, at bpmNEXT, my own annual BPM technology conference, Oracle demonstrated for the first time in public a Level 3 implementation including FEEL and boxed expressions. One vendor told me recently that seeing that for the first time made them understand how those things worked in a tool, and the light bulb suddenly turned on. Then Trisotech demonstrated their DMN editor that can draw full Level 3 boxed expressions, and promised complete Level 3 execution in a few months. Even Sapiens Decision, known for their own modeling language called The Decision Model (TDM), flashed a bit of DMN and suggested they were working on full DMN. And the Workflow Management Coalition, another BPM standards organization, proposed an initiative to help all DMN vendors with tools, test suites, and code for Level 3 implementation.
The wheels were suddenly turning! Here at this conference you are seeing another step forward since April. And after this event, I believe other tool vendors may begin to question their original partial implementation strategy, which only works when no one is doing the real thing.
To understand DMN’s promise, we need take a closer look at the problem it is intended to solve. A decision model describes the logic of a business decision in terms of a set of rules acting on the decision inputs. Given specific values for the inputs, the decision logic determines an output value, the outcome of the decision. That’s all it does. Some wish it went farther. For example, Jacob Feldman is always saying DMN should allow you to define constraints on the inputs and then find the input set, and possibly the ruleset, that, given some goal metric, generates the optimum output value. Now that’s a really interesting problem, but that’s not what DMN tries to do. DMN just says, given a particular set of input values, what is the output value? I admit that is kind of pedestrian, but in fact it is the logic that determines whether your mortgage is approved, the price you pay for insurance, and whether you get that business class upgrade. So decision modeling comes down to defining the inputs and the decision rules, and their dependencies on other inputs and rules.
Decision management for many years was called business rule management. The change in terminology, and DMN’s strong association with that change, signifies a break from the traditional practice of business rule management, which asked the business to start by hunting down and harvesting all the rules buried in application code, in policy manuals, processes, and in the heads of subject matter experts, so they might be centrally managed and executed on a business rule engine. The harvested rules are written down in the form of text-based business requirements, which are given to programmers for translation into a rule language specific to a particular business rule engine. But text-based business requirements are notoriously vague and internally inconsistent, resulting in user acceptance problems, and so the cycle repeats. And that’s not the only problem. Over time, in decision-intensive businesses like financial services, the organization will accumulate thousands of rules, so simply managing them, refactoring them to deal with changing business requirements becomes extremely difficult.
Hopefully you see some of the problems here:
- If the goal is implementing decision logic, harvesting business rules is not an efficient way to start. Most of the harvested rules are not relevant to the decision of interest.
- Text-based business requirements documents lack the ability to ensure rules are complete and consistent.
- Tool-specific rule languages create vendor lock-in and higher cost of maintenance, support, and training.
- Translating business requirements into program code means Long development cycles and high project failure rates that work against today’s key imperative, business agility.
A decade ago, a new idea called decision management began to challenge this traditional practice. James Taylor’s Decision Management Manifesto introduced the “Decisions First” mantra, attacking the practice of starting by harvesting all the rules. Instead, he said, business users should start by understanding the decisions they want to make and the information they require. The rules follow naturally from that.
Then Barb von Halle and Larry Goldberg of KPI, now Sapiens Decision, introduced a business-oriented, but rigorous and systematic, approach to decision management that replaced text-based business requirements with decision models based on a strict set of formats, principles and guardrails, verifiable for completeness and consistency, and more easily maintained. This discipline, called The Decision Model or TDM, emphasized the fact that business users, given proper tools and methodology, are indeed capable of defining and managing complex decision logic themselves, even in large-scale production environments, and this approach proved successful in dramatically improving business agility.
TDM diagrams focused on the logic of individual decisions, or what DMN would call an individual decision service. In his book Knowledge Automation, Alan Fish of FICO introduced the notion of end-to-end decision requirements diagrams (DRDs) representing business decisions executed in multiple steps, including external decisions and human decisions.
Ideas like these were successful in improving the cycle time and success rate of decision management projects. But none of them addressed the proprietary, tool-specific nature of decision modeling. That requires a standard language, independent of any tool vendor or consulting company. Thus was born DMN.
DMN stands for Decision Model and Notation. The “and” is an OMG thing. It means that in addition to a graphical notation, DMN provides a formal metamodel that precisely defines the diagram elements, their attributes and behavior, and a schema for xml serialization, so that the model is not just an outline of the decision logic but a complete executable form of it.
Think about this for a minute. You have a decision model, a diagram of end-to-end decision logic, composed of multiple decision nodes, each with some tabular definition of its rules… And this model not only communicates the logic visually to the business but is actually executable on a DMN engine! All the logic defined visually through diagrams and tables is captured in computable form, as XML. The visual decision model and its XML serialization are equivalent. You can generate one format from the other.
The XML representation of those diagrams and tables can be executed on a decision engine. So… like BPMN, DMN embodies the dream of a common language shared by business and IT. The decision logic, end to end, even if it requires multiple steps, can be defined by the business people themselves. Because that logic is model-based, it is verifiable for completeness and consistency at design-time. And here is the best part: the meaning of the diagrams and tables does not depend on the tool that created it. It’s defined in a spec. The formats and syntax of the language are both tool-independent and executable. Provide it with values for the inputs, and the engine can determine the value of any decision node in the DRD. For high volume production, of course, you could also map the DMN XML to the rule language of a commercial rule engine.
Now.. you can have all the diagrams and tables you want, but to make them executable and tool-independent, the text in those diagrams and tables requires a defined syntax, i.e., a standard expression language.
Now we’re getting to the heart of the matter. Because that expression language needs to be both business-friendly and at the same time rich enough to handle real-world decision logic. The original DMN task force struggled with this. Some were against any formal expression language. Some wanted their own rule language. In the end, Gary Hallmark of Oracle won the day with his contribution, a brand new expression language called FEEL, and a tabular format for assembling those expressions into something like program statements, called boxed expressions.
So DMN, as defined by the spec, is really 5 things rolled into one:
- Decision Requirements Diagrams
- Standard formats for decision tables
- A standard expression language, FEEL
- Standardized tabular formats for all decision logic, called boxed expressions
- XML serialization based on a formal metamodel
But partial implementation, Conformance Levels 1 and 2, don’t require all 5. Conformance Level 1 says you just need #1 on this list, the DRDs.
Conformance Level 2 means DRDs and decision tables conforming to S-FEEL S-FEEL is very simple, just basic arithmetic expressions and comparison tests. It can’t do string manipulation or date-time arithmetic or table queries, but its logic can be verified for completeness and consistency, and has the possibility of being mapped to tool-specific rule languages for more advanced decision logic. So It’s a lot better than Level 1 but still not the real deal.
Conformance Level 3 is the real deal. But there is a complication. DMN does not require FEEL as the expression language; it’s just the default. Technically a tool could use Java or something else. That’s allowed. But… conformance level 3 is specified in terms of FEEL only. If a tool implements some other language, the meaning of the model is no longer tool-independent.
I mentioned at the beginning that I’ve seen that movie before, with BPMN. BPMN was originally just a diagramming notation, not executable. Around 2005, BPM vendors like IBM, Oracle, and SAP thought that was OK because BPMN models could be translated into a similar process execution language called BPEL. By 2008 it was clear that customers did not agree. Even though BPMN and BPEL were similar, the languages were not the same. Certain things you could draw in BPMN you couldn’t execute in BPEL, and vice versa. The whole magic of BPMN’s promise to the business – that what you model is what you execute – proved false, and it was the beginning of the end for BPEL in BPM. So IBM, Oracle, and SAP rushed to develop BPMN 2.0, which unified the diagramming notation with execution.
And that worked. With BPMN 2.0, what you model is what you execute. It transformed the BPM software industry. IBM for example, the number 1 vendor in that space, essentially walked away from its own BPEL based offering in favor of tiny BPMN-based acquisition. Suddenly you had startups offering new BPMN tools, new open source implementations. BPMN 2.0 utterly changed the industry landscape.
Now DMN is trying to do in one step what it took BPMN almost a decade to accomplish. Is that too much too fast? Maybe so. Long term, however, I am convinced that the need to translate DMN models into some other language for execution is not viable.
Some vendors possibly fear that if DMN Level 3 runtimes are widespread, they will be commoditized out of business before the market gets off the ground. I disagree, because even Level 3 leaves a lot of business value outside the scope of the standard. DMN does not include any of these elements of a so-called “whole product”:
- A graphical interface for modeling data
- a business glossary
- A way to incorporate constraint rules
- Anything to do with methodology, or governance
- Anything to do with testing and analysis
- Anything to do with execution, access to physical data, fault handling, or performance optimization
Real DMN tools are going to compete, and succeed or fail, based on those things, even if an open source executable FEEL library becomes freely available.
With that as background, let’s see how DMN as a language works…
First, what is a decision? As we said, it’s determination of an output fact value from a given set of input fact values, where a fact, in DMN terminology, is a variable with a specified domain of allowed values. In DMN, each decision is comprised of a variable that holds its output value, and a value-expression that contains the logic that determines that value. DMN’s value expressions are based on decision rules: if condition1 is true and condition2 is true, then the output value is X. Decision rules are a class of business rules, but not all business rules are decision rules. For example, constraint rules are not decision rules.
Decision requirements. Going all the way back to the decision management manifesto, we know that decisions require information in the form of source data and supporting decisions. DMN describes these dependencies in the form of a Decision Requirements Graph or DRG, the complete network of all supporting decisions and input data required by any decision node. The visualization of a Decision Requirements Graph, is called a Decision Requirements Diagram or DRD. Rectangles in the DRD are decision nodes, ovals are input data. The value expression for any decision node may only reference other DRD nodes directly connected by information requirements, the solid arrows. So the decision Affordability rating here only depends on the variables Applicant and Monthly payment.
A DRD can show additional information. The rectangles with clipped corners, called business knowledge models or BKMs, represent reusable bits of decision logic invoked as functions by one or more decision nodes. The wavy bottom shapes called knowledge sources are effectively annotations describing the basis for the logic, such as policies or analytical models.
BKMs, it turns out, are somewhat controversial. The way they are represented in the diagram, displaying both the calling decision and the called BKM, is not popular with implementers, and many tools do not support them at all. The spec, however, requires a BKM for reusable or iterated decision logic.