What Makes BPMN and DMN Standards?

A couple weeks ago my attention was called to a LinkedIn post, one of those clickbait polls: “Is BPMN the standard? Do you use it? Would you expect it from others?” About 65% said Yes, but naturally the haters clogged the comment thread. Of most interest to me was a comment by Alec Sharp, a respected process modeling consultant, who says…

BPMN is what I call a “claimed standard”… in practice rarely followed, at least as intended. I did an informal survey some years back with some of the top business process folks in the world, and none could think of a single case where BPMN contributed to a significant piece of process analysis and design.

That may be true, but this statement betrays a fundamental misunderstanding of what standards are and what users should expect from them.

Item 2. At this year’s Decision Camp, a technical conference for the decision management community, a presentation entitled “DMN On-Ramp: A Pragmatic Approach to DMN Standardization” proposed a new “consortium of practitioners, stakeholders, academics and vendors” that aims to “provide a ‘staircase up the compliance mountain’ for users and vendors; educate, raise awareness about DMN and its business use cases; [and create an] incremental, vendor neutral route to standardization and compliance.” Admittedly, DMN has underperformed in the marketplace, and a community of practice designed to increase adoption is a great idea. But with DMN there is no “compliance mountain”; the requirements for Level 1 conformance are barely a molehill. It’s true that I have called some tools claiming conformance “fake DMN,” but it hasn’t seemed to bother those vendors. The striking thing about this new organization, which is still not off the ground, is that what it dislikes about DMN are the things that makes it a standard.

In the world of software, a standard provides a functional specification for some component or system, defining the name, semantics, and operational behavior of each element.  It is typically filled with SHALLs and MAYs, distinguishing absolute requirements from nice-to-haves.  The specification is developed and maintained not by a single company but by a group of vendors and practitioners representative of that community.  The group’s procedures are governed by strict rules ensuring vendor-neutrality – adoption of some clause is typically by majority vote – and the resulting intellectual property is available free of charge.

Business modeling standards like BPMN and DMN in addition define a graphical notation – standardized shapes and symbols linked to the semantic elements – and a model interchange format in some universal language such as XML. Standardizing the semantics is an absolute must; without that, you cannot legitimately call it a standard.  In business modeling standards from OMG, the semantics are defined by a combination of a UML metamodel (including its companion XML schema) and tables and text in the specification document.  Without adherence to the required notation and semantics, a product cannot rightfully claim conformance to the standard.

Software products conformant to the standard should enjoy some measure of interoperability, benefitting both end users and product vendors:

1. End users have a choice of product vendors conformant to the standard, reducing vendor lock-in risk.  They might start with vendor A and switch later to vendor B with less disruption than starting over completely.  Or business users might use a tool from vendor A and developers a different tool from vendor B.
2. End users have multiple sources of information about the standard and how to use it effectively, in the form of books, online articles, conference presentations, and independent consultants and training.  They are not solely dependent on their product vendor for that.
3. Vendors can more easily incorporate or interact with components from other providers.  They are not obligated to create an entire whole-product ecosystem themselves.
4. In the case of business modeling standards, vendors can more easily engage business users in defining requirements or even full solution implementations.  This benefit has been a particularly significant factor in the adoption of BPMN and DMN.

If you’ve always suspected that standards, while purportedly for the benefit of end users, were actually created by vendors for their own commercial benefit, you’d be right.  Creating standards is a lot of work, as I know from personal experience.  Members of the task force spend several hours a week in conference calls, developing and testing proposals, and reviewing specification text, and someone has to pay their salary for that time.  Actually, the benefits of a standard accrue directly more to end users than to vendors, who typically must modify their existing products to conform to the spec.  But as RFPs and informal product research increasingly add standards compliance as a requirement, vendors are pushed to at least claim support.

Much of the confusion around standards comes from several factors:

1. A standard’s specification is the work of a committee.  As a result, it typically contains some elements of little utility, or none at all, that were put forward by some task force member and managed somehow to be adopted.  Not all parts of the spec are used in practice.
2. The spec is not a teaching document for end users.  It is written for “implementers”, meaning tool vendors.  The smarter ones find a way to ignore the “junk” elements.  Free posters on the web displaying all elements defined in the spec are not “proof” that the standard is too complicated to use; they are just free stuff on the web.  Methodologies like BPMN Method and Style show how to do just about anything you want with a small subset of elements.
3. Standards are based on a particular conceptual framework and set of principles, ideally one already in common but imprecise use.  But adoption of that framework or principles is not the same as conformance to the standard.
4. Standards sometimes claim an intended audience or pattern of use, in particular with regard to empowerment of business users.  Tools based on the standard may or may not choose to support that pattern of use.

In the case of BPMN and DMN specifically, items 3 and 4 above are at the heart of the controversy.

BPMN originated back in 2002-2003 at the dawn of a new technology, web services.  Before then, not only were there no standards for BPM tools, but human workflow and application integration required separate and distinct technologies.  The startup Intalio created the Business Process Modeling Language (BPML) that used web services to combine human and automated tasks in a single process model.  That was revolutionary in itself, but they went further.  BPML would be created graphically by non-programmers using a standardized flowcharting notation called BPMN.  Swimlane flowcharts had been in common use by the process improvement community since the 1980s, and basing BPMN on that conceptual framework was intended to attract the interest of business users.  It worked!  There was even a book written about it, BPM: The Third Wave by Fingar and Smith, that created a mild sensation back in 2003.

But Intalio was actually too early, as the key web service standard WSDL was not yet final, and when it came out, BPML was incompatible with it.  Instead BPML was replaced by a similar process automation language called BPEL from IBM and Microsoft.  Those vendors had no interest in business user empowerment and hence provided no graphical notation.  But BPMN remained popular with business users, so process automation vendors continued to use it, even without BPML.  Increased availability of inexpensive – sometimes free – BPMN tools made it especially popular among business users looking simply to document their processes, with no interest in automation at all.  Over the past 15 years, I’ve trained nearly 10,000 users on process modeling with BPMN, and well over 90% of them have zero interest in automating their processes.

For automation vendors, the original promise of BPMN to business – What You Draw Is What You Execute – was not really true in the BPEL era.  But in 2008-2009, IBM, Oracle, and SAP needed a way to make it true in order to gain business support for large-scale IT projects based on Service Oriented Architecture (SOA).  The result was BPMN 2.0, which made the BPMN notation a true standard, maintained by a real standards organization (OMG), with formal procedures, well-defined semantics and operational behavior, a metamodel and schema, the whole thing.  The semantics and operational behavior had to be well-defined because now BPMN models were automatable on a runtime engine.

Support from the major software vendors increased the availability of BPMN tools, although the vast majority of BPMN users still had no interest in automation.  And the spec was even more complicated than before, still with many junk elements.  That was the birth of BPMN Method and Style, my methodology for getting rid of the junk and creating models that reveal the process logic clearly from the printed diagrams alone.  Two-thirds of the BPMN 2.0 spec concerns details “below the waterline” – not visible in the process diagrams but there to capture implementation details in the XML serialization.  Some of those details are necessary to understand the flow logic, and so Method and Style uses certain conventions to surface them in the diagrams.

So, to get back to Alec Sharp’s misunderstanding, BPMN as a standard has always been about process automation, not process improvement.  It incorporates a conceptual framework – swimlane diagrams – that long predated it, but adds to that strict semantics and operational behavior behind the shapes and symbols.  That semantic clarity is actually a huge improvement over informal swimlane diagrams, but effective use of BPMN for non-executable models requires combining it with something like Method and Style.  Because the BPMN spec provides a standard XML format for the models, Method and Style’s “rules” can be validated in a BPMN tool like Trisotech or ViziModeler, a huge bonus.

In that LinkedIn post, several commenters boasted about how they use BPMN carelessly or even intentionally incorrectly, but what is the point of private semantics?  Several managers have told me of the vast sums wasted by their BPM project teams because their BPMN models could be understood only by the person who created them.  By respecting the defined semantics of the shapes and symbols and adopting conventions like Method and Style, BPMN models become understandable by all members of the team.

Now let’s turn to DMN, which came out in 2015, several years after BPMN 2.0.  Like BPMN, DMN took as its conceptual framework an idea that was already popular: “Decisions First”, meaning decomposing a complex decision, top-down, into a hierarchy of supporting decisions.  This approach stood in contrast to the older “Business Rules” approach based on harvesting rules from a vast array of legacy systems and subject matter experts, and later assembling the rules, bottom-up, into decision logic of some utility.  What DMN calls a Decision Requirements Diagram (DRD) did not originate in DMN, but had been articulated well beforehand in James Taylor’s Decision Management Manifesto, The Decision Model by Goldberg and von Halle, and Knowledge Automation by Alan Fish.  This is important, because just as swimlane diagrams do not imply BPMN compliance, not every tool supporting the DRD concept is DMN-conformant.

DMN distinguishes decision requirements from decision logic.  Decision requirements, captured as DRDs, are intended to be created by business users.  In that diagram, each decision is represented as a rectangle, with incoming arrows representing the inputs to its internal logic, i.e., other supporting decisions and input data.  Decision logic means the expressions that transform the input values into the output values.  It’s the thing that makes the decision model executable.  For reasons I don’t pretend to understand, DMN took a step that BPMN dared not even suggest, which is that decision logic should be created by business users as well.  I say I don’t understand it because, looking around today, I see few DMN vendors that support that notion… Trisotech being the notable exception.

To enable that, DMN defined a set of standard tabular formats called boxed expressions.  One type of boxed expression, the decision table, was also a pre-existing conceptual framework, but DMN introduced several other new boxed expression types. Boxed expressions are challenging to implement, because tables may be nested inside other tables.  But the real implementation challenge was FEEL, a new business-friendly expression language for the text to go in the cells of those tables.  A major issue for FEEL implementation was that the variables in its expressions are the names of the decisions, input data, and other elements entered in the DRD, even though these contain spaces and punctuation absolutely forbidden as variable names in almost every other executable language.  Even after Red Hat made their FEEL parser and DMN runtime open source, many vendors remained reluctant to support it.

Users have three fundamental expectations of a business modeling standard:

1. Vendor-independence.  The specification is based on a fair and open process with contributions from a broad spectrum of stakeholders.
2. Well-defined modeling element semantics.  The primary distinction between BPMN and the swimlane diagrams that preceded it, and between DMN and the decision requirements models and decision tables that preceded it, is the meaning of each shape and symbol in the standard is precisely defined in a spec.  It is not based on whatever special meaning the modeler or tool vendor has in mind.
3. Some degree of interoperability between tools.  There should be some standard interchange format so that models can be exported from one tool and imported into another.

Without those, you don’t have a meaningful standard.

A major complaint of the DMN On-Ramp group is the “mountain” of conformance requirements.  It’s hard to see this as a mountain.  The DMN spec defines three levels of conformance.  Conformance Level 1 requires support for

• DRDs (Chapter 6), meaning both the shapes and symbols and their semantics and operational behavior;
• Boxed expressions (Chapter 7), although it is not clear which ones must be supported; and
• Decision tables (Chapter 8)

It does not require that expressions in decision tables and other boxed expressions use FEEL, except to say that if the tool “provides an interpretation” of an expression – I think that means it is executable – it must follow the rules of Chapter 7, boxed expressions.

Conformance Level 2 requires everything from Level 1 plus support for a FEEL subset called S-FEEL in decision tables.  S-FEEL is now deprecated, but basically it means decision table cells must be what are called unary tests, extremely simple expressions that allow the table to be verified for completeness.  Basically Level 2 means executable decision tables with DMN-standard syntax in the cells.

Conformance Level 3 requires everything from Level 1 plus FEEL.  The DMN TCK, sponsored by the Workflow Management Coalition, provides test suites of executable models exploring all the obscure edge and corner cases of FEEL, ensuring interoperability of Level 3-compliant tools, a tremendous benefit.

In reality, most tools claiming DMN conformance are something less than Level 1.  They have DRDs, but do not support all the elements and not all the correct shapes.  They typically provide decision tables and literal expressions but no other boxed expressions.  Most important, they do not support FEEL.  In other words, they support the pre-existing conceptual frameworks of decision management, but not the DMN standard.

Even though FEEL is not a requirement for DMN conformance except for Level 3, promoting alternatives to it appears to be at the heart of the DMN On-Ramp effort.  As articulated on their website:

Established vendors already have fully defined expression languages, and ways to manipulate data. For various reasons, it is not feasible for the majority of vendors to replace their implementation with all that DMN defines.

The question becomes this: If tools use their own expression languages and equivalents to boxed expressions, what makes the resulting language – beyond the DRD – a standard?

• Are DMN-standard semantics and behaviors still respected?
• What is interoperable between tools?  Is there a standard XML interchange format, and if so, is it compatible with DMN extensions?

If those questions have good answers, the resulting models would meet my definition of a workable standard, and I am all for the effort.  I am not intractably wedded to FEEL.  It is a very good language, powerful yet easy to use.  I have posted previously on its superiority to PowerFX, Microsoft’s expression language for Excel and PowerApps, but I am not opposed to allowing PowerFX in boxed expressions, or plugging in SQL, javascript, or even some vendor’s proprietary rule language so long as it is just an expression language, not a full programming language.

But I believe this problem could be solved easily within the DMN standard by addressing the Conformance requirements.  The DMN Conformance levels, unchanged since DMN 1.0, are not an impossible mountain to climb, but they are admittedly vague and confusing.  Instead of saying a tool must support all the requirements of Chapters 6, 7, and 8 of the spec, it should be more specific.

Here is my suggestion.  Level 1, signifying standardized decision requirements, should require a tool to support DRDs containing a specific list of elements – decisions, input data, business knowledge models, decision services, and knowledge sources – render them with the standard shapes and symbols and endow them with DMN-standard semantics and behaviors.  If the On-Ramp wants to require some higher level decision requirements features, for example certain metadata elements, I have no problem with that.  Level 1 models should be interchangeable between tools using a standard XML format.

Level 2, signifying standardized decision logic in addition to standardized decision requirements, should require support for certain DMN-standard boxed expressions – at a minimum, decision tables, literal expressions, and contexts – using any expression language in the boxed expression cells.  Level 2 models should be interchangeable with other tools supporting the same expression language.

Level 3 would add support for FEEL.

What the On-Ramp people don’t acknowledge is that DMN is a “live” standard.  It remains in a continuing cycle of revision, and changes like this could be added in DMN 1.5.  There would be debate over what is in the levels, but I doubt that the notion of revising and clarifying the Conformance Levels would have much opposition in the task force.  It should happen!

Making DMN more comfortable for a wider group of tool vendors, without losing the attributes that make it a standard, would certainly result in increased adoption by end users.  And that would be a very good thing.