True North

A good entrepreneur has a canny intuition for their True North. I’ve heard this from many good investors.

Personally I’ve always believed it. One of the bases through which I judge my professional contacts is on their decision making ability. Some people seem to always make good choices. Others, faced only with good outcomes, somehow still manage to find a more painful outcome.

True North varies from project to project. It is not a moral or philosophical concept but rather hard earned project focused direction. It becomes particularly evident when people are operating in emergency-mode, when every decision must be in a positive direction and only a finite number of decisions are allowed before survival-or-failure occurs.

Some of my recent work has been focused on the overlap between pharma and clinical development. And this has led to a realisation that True North in these fields is very, very different.

Medics focus on what is treatable.

Clinical practice is highly pressured. Life and death decisions are made in rapid succession. And in some cases no amount of effort will save the patient. We are probably all familiar with the television tropes of the doctor who refuses to stop trying until their team eventually pull them away from the unsaveable patient.

Combined with the stress is the impact of decision fatigue. This is a now well understood phenomenon in which our ability to make quality decisions declines over the course of the day. The biggest factor impacting this decline is the number of decisions which we have already had to make.

Not all of medicine is this extreme.

Differential diagnosis is a methodology which has been developed in response to the fact that some illnesses need early treatment in order to be effective, and of course not all illnesses are treatable. This is the predominant methodology taught in medical schools today.

As a scientist the biggest surprise in my interactions with medical doctors is usually in their unwillingness to engage in speculation or to order confirmatory tests. This is completely in keeping with their training. Speculation requires cognitive engagement, particularly as they must first evaluate my ability to take in the information, and would reduce their subsequent decision making capacity. And confirmatory tests are only justifiable if they would lead to a change in my treatment.

Medical True North is dictated by what is treatable or will lead to differences in treatments.

Pharma development focuses on what is observable.

I had to read the ICH (International Council on Harmonisation) guidelines recently. I was doing a comparison between pharma and medical AI validation methods. These are the official guidelines for the development of new pharmaceutical drugs. They apply in the USA, EU and Japan.

Combining this formal process knowledge with a recent experience of selling R&D software to a top-10 pharma company, I realised that the pharma equivalent of True North is a focus on observable outcomes.

This does not just extend to patient symptoms in a final clinical trial. Rather it begins there and extends backwards through the development process. At every stage, there are highly trained scientists working in an industrialised process. Each has inputs which they typically accept, from scientists earlier in the pipeline, and outputs which they produce, for the consumption of those later in the pipeline.

This means that when you are trying to sell to pharma you must map to existing inputs and outputs. Using non-accepted inputs or outputs will require a massive process change on their part. It’s ok to bridge scales, skipping a current step, but you must fit into the machine. The process required to develop acceptance of new observables is long and should stand independent of any sales process.

Clinical efficacy and safety are the observable endpoints of drug development.

AI is complicated

In comparison to these True North’s AI development is a complicated story.

On the one hand, a good ML engineer learns early that the only task they have is to optimise for the objective function.

On the other hand, if the objective function does not align with the use case then the tool under development is not use to anybody.

ML engineers persist in their simple true north as it simplifies their life a lot. It also takes an expert to point out that the objective function did not match the fuzzier real-life objectives of the use case.

From my perspective there are three different levels of True North in AI development

• Technical / near term: fit the objective function, or you will be fired.
• Use case / mid term: build a tool which has real impact, or nobody will use it.
• Scalable / long term: this is where things get hard, unless your ‘AI’ will provide a scalable application then there are probably cheaper approaches than using ML.

You can go very far in developing AI before anybody can correctly call you on the scalability of your solution. This is why so many people promise this but only a few companies regularly deliver.