At the recent Gartner BPM Summit, I was shocked to see how high a pedestal the Gartner analysts now place simulation analysis in their gallery of must-have BPM capabilities. Ever obedient, the BPMS and modeling tool vendors now universally throw it into the box. How else to get into that Magic Quadrant? But have these analysts ever really used these tools, or even scrutinized them closely? I’m not really sure. I haven’t looked at all of them myself, but my sampling to date tells me this is a fake feature if ever there was one. In my training Process Modeling with BPMN, I wanted to add a big section on how to leverage the cool parts of that notation — events and exception handling, delays and batching patterns, etc — to project process performance using simulation. I know simulation isn’t part of BPMN — the spec not only doesn’t standardize simulation parameters but doesn’t even have the barest concepts of resource pool modeling — but it is a nearly universal feature of modeling tools that use BPMN. So that was my starting point. Since the simulation tutorials of the tools I looked at — where they existed at all — didn’t address anything close to real problems of process analysis, I had to make up my own methodology and then apply it to the notation. It was in the course of doing that that I concluded that most modeling tools were missing four basic ingredients needed to make simulation real, not fake: 1. the right simulation parameters Let’s go through this, and if you still think your simulation tool isn’t fake, chime in. I should mention that the tool I wound up using in the training, Process Modeler 4 from itp commerce — one I touted here a couple weeks ago — was not only the closest to what I needed, but the vendor was willing to tweak the features in several places to try to get it right. You can’t ask for more than that. Let’s start with what kind of predictive performance analysis are you expecting from simulation? Two things came immediately to my mind. The first is the kind of analysis most folks have in mind when they document their as-is processes and sit around looking at the diagrams discussing why it takes so long, costs so much, and is so error-prone. Classic BPM “theory” says that the problem is in the handoffs, typically between roles or departments (swimlanes in BPMN). The improved to-be model has fewer yo-yoing handoffs between swimlanes, less unnecessary rework or pointless reviews and approvals. Thus, simulation use case 1 is to quantify the improvement. How much faster, how much less cost, etc? In this kind of analysis, remember, the problem isn’t about contention for resources — too few widget-tweakers at step 17 — but about inefficiency in the structure of the process itself. The second is the kind of analysis that most BPMS vendors promote, particularly those focused on workflow automation. There it’s all about “bottlenecks” — too few widget-tweakers at the widget-tweaking step — possibly exacerbated by the fact that the same pool of widget-tweakers is also supposed to handle the widget-flipping task at step 19. We call this “contention for resources.” There the simulation is about creating various what-if scenarios with different numbers of widget-tweakers to see the bottlenecks disappear. You don’t need to be a rocket scientist — you could even be a widget-tweaker — to know that if you configure enough widget-tweakers the bottleneck at step 17 will go away. So what? The issue is at what cost. So this analysis is about looking at the cost/time tradeoffs. There is a variation of this type of analysis called activity-based costing. Here the idea is that different classes of work running through the process incur different costs. Simulation analysis by itself only gives you the direct or variable costs of each type. Activity-based costing lets you distribute the fixed overhead and other indirect costs as well in a “fair” way. In the newer forms of the analysis, called time-driven ABC, you can get everything you need from the simulation output, so it’s really an extension of the previous resource utilization use case. So both of these simulation use cases (or all 3, if you count ABC separately) are important. But they’re different. The way you have to set up the simulation parameters is different, the standard outputs are different, and the custom reports you need are different as well. I worked through all 3 of these in my BPMN training, but my post here is about the simulation tools, not the training. In the interests of time, here are the features I discovered I wanted in order to do all three. We can get into why later, if there is interest. 1. Activity duration parameters. One parameter is not enough. You need at least two, the active time (consumes resources) and a wait time (does not). And once you have all this stuff, you need to show business process analysts how to use it, cookbook-style, for the three simulation use cases I described, maybe others. I could go on, but you get the point. You say your simulation tool doesn’t provide many of these things? That’s because it’s a fake feature.
2. the right calculation of standard metrics
3. the raw instance data needed to create custom metrics and reports, and
4. a methodology for translating the kind of performance analysis that’s supposedly possible into the parameters and outputs provided.
2. Event parameters. If your diagram has BPMN intermediate events, you need to provide parameters describing both the probability that the event occurs and the time it occurs. Ditto for event-based gateways. And if your tool doesn’t support these events and event gateways, I’d say it’s useless anyway.
3. Cost-per-use parameters for certain activities, in addition to the normal cost-per-hour.
4. Shift calendars for all resources, human and machine.
5. Priority and priority aging logic for resource contention. Which activity gets the resource instance? Which process instance gets the resource? This doesn’t affect the mean as much as the distribution of costs and times.
6. Contention for resources across multiple processes. (My tool didn’t have this.)
7. Breakdown of metrics at process, activity, and resource level.
8. Histograms of cost and time. Means are almost useless. It’s the outliers you need to worry about.
9. Resource utilization rate calculation.
10. Raw simulation data at the activity-instance level, ideally dumped into a tool like Excel that you can manipulate easily. Creating Excel named ranges? Even better! You can’t really do simulation analysis without this data. The standard reports are never exactly what you want.
11. Some rudimentary form of instance-level properties used by the simulation engine to identify types of work, and the ability to define simulation parameters as expressions of these properties. My tool doesn’t have this. It’s a slippery slope from instance properties to a full data model and a shift from simulation to emulation and debugging. We don’t want to go down that slope. Well maybe just a little. To do anything with quality metrics (not just cost and time) I think you need instance properties.
Is Simulation Fake?
[This is a re-post of something I wrote yesterday on the SAP Business Process Expert megablog, in case you don’t follow that site.]
Share This Story, Choose Your Platform!
5 Comments
Leave A Comment
You must be logged in to post a comment.
Maybe you’re using “fake” modeling tools, Bruce. Typically those tools that just produce drawings don’t and can’t do simulation because it takes a lot more information and more rigorous diagramming to produce any useful simulations. I for one have been using modeling tools that are capable of simulations for 10+ years and there are several tools that do. I agree with you if you are implying that unless a modeling tool is capable of serious simulation and analysis it really doesn’t serve the needs of a serious business process analyst/designer. The product that I’ve been using for the last several years can meet all of your criteria and more – Corporate Modeler Suite by Casewise. Of course, the tool isn’t really the issue anyway. The issue is whether or not you are working in an organization that will support the extra effort to develop rigorous models that let you experiment and have some degree of confidence in the designs. The real problems with doing simulations are that too often, the data necessary to create them doesn’t exist and has to be “discovered” and that there are too few analysts who can create valid simulations and too few managers who understand what goes into it and the value it provides.
Dr. Bruce Silver,
My first simulation project was back in 1973 at the Ministry of Transportation and Communications of Ontario, Canada, when I was a Computer Liaison Officer. I used GPSS (General Purpose Simulation System) from IBM to construct and run several simulation models for testing and implementing a computer-assisted and Operations Research based Strategic Planning System. That system was a component for implementing the PPBS (Planning, Programming, Budgeting System), which all Federal and Provincial/State governments use since 60s or 70s. DoD was the first department to use PPBS. President Johnson mandated it for every US Federal departments and agencies in 1965. The Province of Ontario adopted it since 1970.
Simulation is based on a well defined activity model and sampling of real-life performance statistics. The statistical data would be analyzed with regression analyses to pick the best-fit probability distribution. Real-life data is used as a guide to calibrate the simulation model by running many iterations through a large prime number as the seed for random number generator. This is to ensure that the model does reflect the reality. The parameters of the model are then varied to test for sensitivity (i.e. ‘if scenarios’). For process improvement, a new model is constructed, tested, and compared to the same set of real-life statistical data. In most cases, the activity model is constructed from several queueing models. In general, the arrival pattern follows the Poisson Distribution while the service pattern follows the Negative Exponential Distribution. For telecommunications network simulation model, the service pattern would be Erlang distribution (a specialized form of Negative Exponential Distribution). When I was on contract with Cantel (now Rogers), a cellphone service provider, as the Chief Architect in 1988 and 89, GPSS was used to assist in the design and development of cellular telephone network across Canada from coast to coast!
I don’t see how PBMN and BPMS, in their current forms, can really support simulation as I define above. It should be noted that I don’t speak from conjectures, I speak from experiences as a practitioner of OR (Operations Research). Moreover, several BPM simulators use Triangular Distribution (which is easy to program but doesn’t reflect reality) and Monte Carlo method for simulation. Monte Carlo method is used for evaluating ill-defined mathematical functions through Variance Reduction, not for simulating processes!
I hope that your Ph.D. would help you to understand what I have said above.
Best regards,
Francis Ip
[…] Bruce Silver, an independent analyst who focuses on BPM, wrote a pretty interesting post about simulators back in March of this year called Is Simulation Fake? . […]
Well said. People think, since the process diagram is ready, simulation is the next easy step. Actually data needed for simulation is very tedious job. I don’t think BPM is designed with simulation in mind
[…] of BPM suites (think Gartner’s magic quadrant). While part of the problem might be the actual simulation tools provided with BPM suites, one has to ask the obvious question: if business process simulation makes […]