Haute Lumière

Commerce · I.05 · MMXXVI · daylight

La Bourse  /  Volume I  /  Nº I.05  /  Workbook — the executive

WORKBOOK — THE CORPORATE EXECUTIVE

Chapter I.05 · The Pilot That Pays

For the person with a P&L, a signature limit, a board and a quarter. This workbook uses the language of the firm without apology, because the firm's own estate already contains the comparison you need — it has simply never been arranged as evidence.

---

THE PREMISE, STATED COMMERCIALLY

You are being asked to do one thing: stop buying pilots and start underwriting a portfolio of them.

A pilot is an information-acquisition cost. Priced individually it is a bet, and your capital committee is correct to treat single bets sceptically. Priced as a portfolio with a measured success rate, a measured realisation rate and a measured verification cost, it is a rate of return — and at the figures in this chapter it is a rate of return of 27.7 percent against a 9 percent WACC, with a break-even success rate of 19.5 percent.

Fewer than one pilot in five needs to work.

That sentence is the proposal. Everything below is how to produce the evidence that makes it true in your firm rather than in an example.

Two things stand between you and that number, and both are decisions made in week one that cannot be made in week twelve. The first is statistical power — most corporate pilots are incapable of detecting the effect they are looking for, and report failure when they have simply been unable to see. The second is the realisation rate — most measured savings never reduce a budget line, and a saving that does not reduce a budget line cannot be spent, reinvested or claimed.

---

PART ONE — DISCOVERY

Days 1–20: the comparisons your estate already contains

Exercise 1.1 — The natural-experiment sweep (one week, with your analytics lead)

Do not commission a study. Spend two hours finding structures you already own.

1. Multiple comparable units. Depots, branches, plants, wards, stores, crews, regions. Pull the list with their monthly outcome history. Four units doing the same work under different management are a control group that costs a query.

2. Queues and waiting lists. Anything where demand exceeds supply and allocation is currently made by judgement. That allocation can be randomised among equally eligible candidates at no cost and no loss, and it produces evidence for free. Oregon did not set out to run an experiment; it ran out of Medicaid places and drew a lottery.

3. Rollouts already scheduled. Any programme reaching sites in some order. The order is currently set by convenience. Randomise it. Nobody is denied anything — the change is the sequence, not the access — and you have a stepped wedge, which is the single most politically acceptable experimental design in existence.

4. History. Thirty-six months of monthly data on the outcome you intend to change. It is the cheapest statistical power available to anyone and it is already paid for.

5. Near misses. A site scheduled for a change that did not receive it for a reason unrelated to the change. That is a natural experiment, often better than one you could have designed.

Output: one page listing what your estate contains, with the number of comparable units and the length of usable history against each.

Exercise 1.2 — The series audit (half a day, with your controller)

For the three outcomes you most want to move, establish:

• How far back the series is comparable. Note every definitional change, system migration, reorganisation and acquisition. A break in the series is a break in the evidence, and it is better found now than in the results meeting.
• Who owns the definition. If nobody does, that is your first finding and it is worth more than the pilot.
• The period-to-period standard deviation σ, and the lag-1 correlation ρ.

These two numbers determine everything you can and cannot learn this year. Most firms do not know them for any of their principal operating metrics.

Exercise 1.3 — The precedent inside your own firm (2 hours)

Find a past pilot that was declared a success and scaled. Pull the file. Ask three questions:

1. What was the comparison, and could it have detected the claimed effect?
2. Did the claimed saving appear in a subsequent budget?
3. Who verified it, and to whom did they report?

Do this appreciatively, not forensically. You are looking for the pilots that did hold, and for what made them hold — and you will find that the ones that held had a named budget owner from the start.

---

PART TWO — THE ARITHMETIC

Days 21–40: the two numbers that decide the year

Exercise 2.1 — The minimum detectable effect (half a day)

For each candidate intervention, compute what your design could see.

  detectable effect  =  2.8016 · σ · √(1/m + 1/k)
  n per arm          =  15.70 · (σ / δ)²
  design effect      =  1 + (people per unit − 1) × ICC

Work through it in this order, and write each number down:

1. σ from history.
2. Inflate for serial correlation. At ρ = 0.5, twelve months carry the information of 4.5 independent ones — a 63 percent overstatement of precision if you skip this.
3. Choose m and k. Buy pre-periods first; they are free. One before and one after sets the bar at 3.96σ. Twelve before and three after sets it at 1.81σ — a 4.80× gain in equivalent sample size, for an email.
4. Apply the design effect if you are randomising units rather than individuals. Thirty people per unit at ICC 0.05 multiplies your requirement by 2.45.
5. State the MDE and compare it to the effect you believe in.

The decision this produces. If the MDE exceeds the effect you believe in, you have three honest routes and one dishonest one. The honest routes: lengthen the pre-period; choose an outcome with less noise; or add units. The dishonest one is to proceed and hope. Proceeding and hoping is how a firm spends £120,000 to acquire an anecdote.

Exercise 2.2 — The claim map (one day, with the budget owner)

This is the exercise that separates a pilot that is admired from a pilot that pays. For every pound of expected saving, fill one row:

Then compute the realisation rate honestly:

      φ  =  banked saving / measured saving

And apply the step test. Costs come in units — a person, a shift, a vehicle, a lease, a licence band, a contracted volume. claimable = step × floor(saving / step). A 55 percent reduction in a shift's work banks nothing. A 110 percent reduction banks one whole shift and φ jumps from zero to 0.91.

The design rule: size the pilot so its saving crosses at least one whole claimable step, and name the step in the proposal.

If no achievable pilot crosses a step, take one of three routes: aggregate several savings that land on the same step; label the work a feasibility study with no financial claim; or choose a saving on a continuous line — energy, consumables, freight, contracted volume — where every unit counts. All three are better than the proposal you had, which is why you want to discover this on day thirty.

Exercise 2.3 — The regression-to-the-mean check (1 hour)

Look at where you were about to run it. If the site was chosen because it is struggling, compute what it will do on its own:

  free 'improvement'  =  (1 − reliability) × (distance from the mean)
  10th percentile, reliability 0.6   ->   0.51 σ

Half a standard deviation of counterfeit success. Either randomise the assignment, or select on a period you then exclude, or keep a control group — which absorbs the regression identically in both arms and cancels it.

Exercise 2.4 — The portfolio inequality (30 minutes)

The number that belongs on the front page of your paper:

          p · φ · S
      -------------------   >   WACC
        C  +  V  +  A

  0.60 × 0.65 × 96,000 / (120,000 + 9,000 + 6,000)  =  27.7 %   vs  9 %
  break-even success rate:  p = 19.5 %

Use your own C, V and A immediately. Use the chapter's p and φ only until your first two completed pilots give you measured ones — and then never again, because a measured φ from your own firm is the single most persuasive figure in the entire proposal.

---

PART THREE — DREAM AND DESIGN

Days 41–60: build the instrument

Exercise 3.1 — The pre-registration, in corporate form (2 hours)

One page, dated, two signatures, lodged with finance and with the named verifier before deployment.

1. The question, in one sentence.
2. The primary outcome — exactly one, defined so that finance and operations compute it identically.
3. Secondary outcomes, listed now and labelled secondary.
4. The comparison: which units treated, which not, how assigned.
5. The periods: m before with dates, k after with dates.
6. The exclusion rules.
7. The failure condition — what result would cause you to stop.
8. The verifier, named.

Item seven is the item that changes the meeting. An executive who has written down in advance what would change their mind is believed about everything else on the page, and this is the cheapest credibility available to you in any quarter.

Exercise 3.2 — The governance split (1 hour)

Three roles. The operator and the verifier may never be the same person.

• Sponsor — carries the decision and the budget.
• Operator — runs the intervention and wants it to work. This is not a fault; it is the condition of getting anything done.
• Verifier — named before the result is known, works to the pre-registration, reports to finance rather than to the sponsor. Internal audit is sufficient and is usually free. Name IPMVP as the reference method where a standard is wanted — Option C for whole-facility comparison, Option B where the affected system can be metered directly.

Write the three names down. If one name appears twice, the result will be defensible to you and to nobody else.

Exercise 3.3 — The board paper (half a day)

Four pages, in this order, and no other order:

1. Page one — the portfolio inequality, with your numbers, and the break-even success rate. One page, one number, no narrative.
2. Page two — the design, including the MDE and what it means: this design can detect an effect of X or larger; below that we will report that we could not tell.
3. Page three — the claim map, with the budget owner's signature already on it.
4. Page four — the fund: size, reversion terms, governance, and the regeneration rate you are targeting.

What is deliberately absent: the case for regeneration, the values argument, the sector benchmark and the peer comparison. You are not asking the board to believe anything. You are asking them to underwrite a measured rate.

---

PART FOUR — DESTINY AND DELIGHT

Days 61–90: make it revolve

Exercise 4.1 — The fund's regeneration rate (30 minutes, then quarterly)

Treat the pilot budget as a stock, and the banked savings as its regeneration.

      r  =  φ · S / C            doubling  =  ln 2 / ln(1 + r)

   φ 0.65  S 96,000  C 120,000   ->   r = 0.52/yr   doubles in 1.66 years
   φ 0.40  S 96,000  C 120,000   ->   r = 0.32/yr   doubles in 2.50 years

Put r in the standing pack. It is the one number that tells you whether the programme is a cost centre or a compounding asset, and it takes one line to compute.

Exercise 4.2 — Size the successor to the saving (1 hour)

The successor a first pilot can genuinely fund in one cycle is φS/C times its size — 0.52×, not 3×. Ask for 3× and you will be refused once and the ladder stops. Ask for 0.52× and take it four times, and in three years you are larger than the refused proposal would have made you in five.

Write next year's programme as a ladder with four rungs rather than one step.

Exercise 4.3 — The three failure modes, pre-empted (1 hour)

Write one line of prevention against each. These are the three that actually happen.

• The fund is raided. A gap appears elsewhere and the replenishment is swept into it. Prevention: a written reversion clause, a named owner, and a line in the standing pack, so removing it requires an explanation.
• The baseline register rots. Definitions drift, a system upgrade redefines a metric, and two years of comparison quietly become incomparable. Prevention: an owner, an annual review, and a version number on every definition.
• The verifier becomes the advocate. The slowest and most dangerous, because it looks like success. Prevention: rotate the role, and have the verifier's own work read by somebody who did not produce it.

Exercise 4.4 — Delight, deliberately (ongoing)

The adoption mechanism is not the mandate. It is the moment somebody in another division asks finance for the baseline template without being told to.

Make the artifacts good. A one-page pre-registration that is genuinely pleasant to fill in gets filled in. A claim ledger that is legible gets read. A results meeting where the sceptic's objection is already answered on page one is a meeting people want to be in. Design for the second adopter, who has no obligation to you at all.

---

THE QUARTER'S WORK — what exists at day ninety

1. The estate map — comparable units, usable history, σ and ρ for three principal metrics.
2. One power calculation with a stated MDE, signed.
3. One claim map with the budget owner's signature.
4. One pre-registration, dated before deployment, with a failure condition.
5. One verified result, read against the pre-registration by a named verifier reporting to finance.
6. One ledger row — metric, amount, budget line, step crossed, owner, date, share returning to the fund.
7. A measured φ and a measured r for your own firm, which is the asset this whole quarter was for.

---

SELF-ASSESSMENT

The two that matter most are the third and the fifth. The third determines whether the money is real; the fifth determines whether anyone believes you.

---

THE SENTENCE TO TAKE INTO THE ROOM

Not this is the right thing to do. Not our sector is moving. This:

"We are proposing a revolving facility with a measured portfolio return of 27.7 percent against a 9 percent cost of capital, break-even at a 19.5 percent success rate. The first pilot's realisation rate was 0.65, measured, and the analysis was pre-registered before deployment. Here is the claim map with the budget owner's signature on it."

That is not an ethical proposal. It is the cheapest capital available to the organisation, and it should be presented in exactly those words.

---

APPRECIATIVE QUESTIONS FOR YOUR EXECUTIVE TEAM

1. Which of our past pilots actually held, and what did the ones that held have in common that the others did not?
2. Where in this estate do we already own a comparison we have never used as evidence?
3. If every rollout here were staggered in a randomised order as standard, what would we know in three years that we do not know now?
4. What would make an operating unit glad to return a share of a saving to the fund, rather than resigned to it?