This is my third attempt, over the course of 9 months, to write this article. The first attempt foundered on my desire to go into detail on whether explanation or explanability is a good characteristic of a model or not. I confess, this was overly motivated by my personal frustration at having worked with somebody who, “never let the facts get in the way of a good story.” The second attempt got lost in a forest of anecdotes from previous projects. I was trying so hard to knit them together that I failed to make a point. Today, I want to focus on the single most important thing that I have learned about developing decision making models.
What is the point?
I am glad you asked. Models always have a purpose. Technical workers frequently get so worried about the technical difficulties in implementing a model that they often forget the marco-purpose of producing the model.
Model building is hard. Let’s acknowledge that for an instant. The technical skills required to build – optimise – predict are significant and usually entail many years of prior training. Of course, this means that the specifics are highly path-dependent. The year the modeller began their studies strongly influences the style of models they can make and how they typically focus their energies.
Beyond technical difficulties there is the domain issue. Building models for a purpose requires that somebody with domain expertise specify the goals of the model. Converting project goals into a model description is an expert process quite apart from the actual implementation of a model on computer hardware. If the project definition comes from a domain expert with no modelling expertise then it is unlikely that they will describe the requirements in terms which a typical modeller will correctly interpret. This act of translation from what is desirable to what is feasible is one of the hardest aspects of modelling. Sadly, it is highly unusual to find an expert-level modeller who also has good domain expertise in more than one domain.
The best kind of modeller, who I can recommend, is either somebody who has years of experience on the specific topic on which the model is to be built, or, they have experience of switching between semi-related fields and have a demonstrated ability to talk with domain experts and correctly interpret their needs. The first person is in some ways the ideal candidate, apart from one small problem, if you know how to accurately describe that person then you are probably able to supervise the building of the model, by a more junior modeller, yourself. If the second type of candidate is additionally capable of working using more than one style of modelling technology, all I can say is hold onto them tightly, they are worth their weight in gold!
In learning how to make mathematical models, the first level is always just to learn correct techniques. For advanced techniques this can take years at graduate school. The second level is being able to appropriately interpret the model. In some cases this means being able to read coefficients and attribute them to real-world phenomena, in others it means being able to discern which applications are beyond the scope of the model. The highest level, of modelling skill, incorporates all of these skills but maintains a constant focus, and often a dialog, on the deployment purpose of the model. The model, as described ‘on paper’ or in code, must match the eventual use-case.
What is it good for?
Models have many purposes. When we study model making techniques there is, perhaps, an over-attention on explanatory models. These are models where each aspect of the model can be related, albeit tenuously, to a real-world phenomenon.
The reason we focus on explanatory models is in-order to motivate students. By using an explainable model in teaching we come up with a reason to learn the relevant techniques which appeals to a wider audience. Some people learn techniques only because they want to understand a physical process better.
However, this means that many expert model makers have worked primarily in environments where the purpose of the model is either exploratory or explanatory. These are excellent aims in-and-of themselves. But they can only serve as inputs to good decision making!
When you read in the news that scientists are complaining that politicians are misusing their models, such as in the public planning process surrounding the recent Covid-19 pandemic, you can probably assume that the scientists in question produced a model with good explanatory powers but it was not fit for the political decision making process. This is an example of a model being used for the wrong purpose.
For complicated explanatory models practitioners are usually focused on technical risks in their implementation. This is typically where their model can be shown to be wrong and so it is where they focus their efforts. However, a decision maker making a decision which incorporates the same model is rarely interested in technical risk. They have an entirely different process around model interpretation. When faced with an explanatory model decision makers typically assume that the model does not truly represent the real-world and thus incorporate its outputs as one of many points in their decision process.
Interim summary
This is the point at which my previous attempts at writing on this topic begin to go seriously off the rails. They are distractions from my main point – models must be made with a purpose in mind.
Over the coming weeks, I will try to publish each of the related topics in a series of shorter posts. My list so far includes:
- Styles of Decision Making models.
- 3 projects I have conducted in recent years. They show the difficulty a modeller is faced with when producing models for other fields.
- Intermediate representations of models. A modeller’s take on what will be a necessary building block for a common language of modelling.
- My favourite least-favourite quote. George P. Box and how he would shudder at how his words are used today.
- A 3-level hierarchy of model building ability.
- Narrative and models.
Why am I interested in this?
It is customary to begin an article with a motivation. I still remember my English classes in school where I would be highly criticised for getting creative and putting the motivation elsewhere. Apparently people find it harder to read. I confess this rationale is entirely true. I have learned it through a lot of reading and a lot of writing, moreover through a lot of rewriting.
I have a form of writers block on this topic. I think so much about modelling and decision making models in particular that I have too many thoughts on the subject. That is why this is attempt 3.0 and the first to actually appear. So I have narrowed the focus enormously and the introduction reflects this.
Here, I want to also acknowledge that I have a far greater motivation about writing about decision making models than just saying that models must have a purpose. I have spent my career working with domain experts to make models. I love making good models. And I have largely avoided projects which involved modelling for modelling’s sake.
I want to improve the quality of model making. I want modellers to read this and up their game. A technical analysis of a model (and I have done many) is only interesting if the model is first interesting. And the model is only interesting if it either gives insights into better modelling or if it accurately contributes to our real-world needs.
Too many of us conduct a model analysis for the simple reason that it is within our skillset. Then we affect a worldview in which our analysis somehow gives great insights into better modelling. It probably doesn’t. However, even the simplest model serving real-world processes probably helps domain practitioners enormously. How about we start there and we only go back to our technical analyses after we have re-learned which models are important to begin with?
One Reply to “Decision Making Using Models 3.0”