Auditing a Governance System

Volume VI — Governance and the Commons

Nine movements, six measures.

THE PLATE

Four colleagues standing in conversation by a wall of windows, the sun low behind them. — Plate VI.10Five Years of Tuesdays.Nobody wrote these to be compared with each other. That is exactly why they can be.

THE LETTER

You have read the constitution. It is good. It has quorum rules, an appeals route, a conflict-of-interest clause, a schedule for elections and a paragraph on how a decision may be revisited. Somebody thought hard about it, probably in a long evening, probably more than once.

That document tells you almost nothing about whether the governance is working.

This is not a cynical observation. It is a structural one, and it is the same observation Elinor Ostrom made about commons: the presence of a rule is not the presence of the practice the rule describes. Her contribution was never that commons work. It was eight design principles that specify the conditions under which they hold — and the meta-analytic record on those principles, which Chapter VI.02 sets out in full, is a record of coding cases, not of reading charters. Somebody went and looked at what people did.

So this chapter builds the instrument the rest of the volume implies. Six measures, computed from documents the body already produces, that together answer the question a constitution cannot: is this governing, or is it describing?

You will also find here the part that makes the instrument honest. Every governance metric worth having is gameable by the people being measured, who are also the only people positioned to collect it. That is not a reason to measure less carefully. It is a specification: it tells you which measures need a second route, which need a protected holdout in the sense of Chapter II.10, and which need to be computed from a record nobody wrote for you.

Six measures, three of them gameable, one instrument that survives all three.

— The Editors

DISCOVERY

What is already working

Governance auditing is not a proposal. It is an existing practice with a publication record, and the best of it is better than most financial auditing.

Ostrom's principles were already an audit instrument, and somebody ran it at scale. Michael Cox, Gwen Arnold and Sergio Villamayor-Tomás reviewed ninety-one empirical commons studies against the eight design principles and found them broadly supported — roughly two-thirds of the studies confirming — while also concluding that three of the eight needed splitting, which is how the familiar eight became eleven. Read that as the achievement it is. A theoretical framework was subjected to a coded, reproducible review by people who did not write it, and it came back supported and amended. That is the condition Chapter II.10 calls plumbing: an international coding protocol, a published method, and independent compilers. Governance research has it. Most organisations auditing their own governance do not.

The forest commons network measures the outcome, not the description. The International Forestry Resources and Institutions programme, founded by Ostrom, sends teams to the same forests repeatedly with the same protocol: biomass plots, user surveys, rule inventories, enforcement records. Ashwini Chhatre and Arun Agrawal's analysis of eighty forest commons across ten countries found that local enforcement and local rule-making authority predicted forest condition, and Lauren Persha, Agrawal and Chhatre showed in Science across eighty-four sites that joint livelihood-and-biodiversity outcomes were more likely where users actually participated in making the rules. Note the design. Nobody asked a forest committee whether it felt participatory. Somebody counted the trees and separately counted who made the rules.

A city published its own participation figures, annually, for a decade and a half. Porto Alegre's Orçamento Participativo did something rarer than participatory budgeting: it kept and released the count. Attendance by region, delegates returned, works approved, works executed, year on year, in a document anybody could pick up. That is what made two decades of external analysis possible — Baiocchi's ethnography, Wampler's comparative work, Avritzer's institutional history — and it is why Sónia Gonçalves and, separately, Michael Touchton and Brian Wampler could later test whether participatory budgeting municipalities in Brazil shifted spending toward sanitation and health and whether child health outcomes moved with it. The audit was possible because the body kept the record before anyone asked for it.

A large cooperative commissioned an audit of its own governance and published it whole. After the Co-operative Group's crisis, Paul Myners conducted an independent governance review and released it in 2014. It did what a governance audit should: it described the actual composition and competence of the board against the actual scale of the business, and it went to the members, who then changed the rules at a special general meeting that August. A membership of millions, a democratic structure, and a published finding that the structure was not functioning as written — followed by a member vote to change it. That whole sequence is the practice this chapter is trying to make routine.

And the people who actually govern commons have always audited practice directly. Irrigation associations count turns taken and water delivered. Grazing commons count stock on the hill. Mondragón's cooperatives count assembly attendance and the ratio between the highest and lowest pay, because both are constitutional matters and both are checkable in an afternoon.

Five cases, one pattern, and it is the same pattern II.10 found in measurement generally. In every one, somebody measured a behaviour rather than a provision, kept the record for its own sake, and made it available to a reader who had no stake in the answer. Where those three hold you have an audit. Where they do not, you have a governance statement, which is a different genre and belongs to a different department.

THE ARITHMETIC

What works, what does not, and where the line sits

First, the six measures, and why counting is the wrong verb for the first one.

The six are: weighted participation; decision latency; reversal rate; the share of decisions taken by exception; concentration in the minutes over a year; and the gap between the written rule and the observed practice. Each is computed from documents a body already produces. None requires a survey.

Start with participation, because the standard treatment of it is arithmetic malpractice. A body reports the number of people who came. That number is a count of attendance, not of participation, and the difference is the whole subject.

Borrow the instrument political science has used since Markku Laakso and Rein Taagepera proposed it in 1979 for counting parties: the effective number, which is the inverse of the sum of squared shares. Give each participant the share of decision weight they actually hold and compute

      N_eff  =  1 / Σ sᵢ²

An assembly of equals returns its own headcount. A body where one small tier holds the decisive vote returns approximately the size of that tier, however many people were in the room.

Run it on Porto Alegre at its height. Roughly 20,000 people a year attended the assemblies, in a municipality of 1,360,590 — a counted participation rate of 1.47 percent. Above them sat around a thousand delegates, and above those the Council of the Participatory Budget: sixteen regions and six thematic assemblies returning two councillors each, a body of forty-four. One councillor for every 455 attendees; one for every 30,922 residents.

Weight the tiers — sixty percent of the decision weight at the council, thirty at the delegates, ten in the assembly — and the effective number of participants is 121.

  counted participants                20,000
  effective participants                 121
  ratio                                  165 x

Two orders of magnitude between what was reported and what was held. And now read the same figure the other way round, which is the reading that matters: Porto Alegre's city chamber has thirty-six seats, so the participatory process ran at 3.4× the effective participation of the elected body sitting beside it, and it did so annually rather than once in four years. Both sentences are true and an audit that reports only one of them is advocacy.

The weights are a parameter, not a measurement, so publish them with the figure the way II.10's parameter register requires. At councillor-heavy weights the answer is 69; at assembly-heavy weights it is 263 — a 3.84× swing on three numbers no dataset contains. The delegate count barely matters: move it from 800 to 1,400 and the central answer moves from 121 to 121.

Second, latency, and the floor underneath it.

Measure decision latency from the first external trace of a demand to the moment money moves. On the published Porto Alegre cycle, a demand raised at a first round assembly on 1 April reaches an enacted budget in 258 days and a completed execution year 639 days later — about 1.75 years.

That looks damning until you ask the auditor's question: compared with what? A conventional capital item raised with a councillor on the same April afternoon enters the same budget law and the same execution year. The latency attributable to the participatory process is zero days. Nearly all of the 639 is the fiscal calendar, which is set by statute and has nothing to do with governance quality at all.

So the honest form of the measure is excess over the calendar floor. Report raw latency and you have audited the constitution of Brazil. This generalises: every governance body sits inside a cycle it did not choose — a budget year, a term, a harvest, a licensing round — and the auditable quantity is the part that is not the cycle.

Third, the denominator, which is where most governance reporting quietly fails. The Orçamento Participativo governed the investment budget. Capital investment is a minority of municipal spending, so between 80 and 95 percent of the money was decided somewhere else, by people the celebrated process never touched. That is not a criticism of Porto Alegre, which never claimed otherwise. It is a requirement on the auditor: state what share of decisions the body you are auditing actually takes, or you have measured a numerator and published it as a rate.

Fourth, the observer effect — and the cut this chapter turns on.

Everyone knows that watching a meeting changes it. The usual response is to watch more meetings. Cost that out before you commit to it.

A single meeting is not an instrument. If a body takes one decision in five by exception, a visitor sees a meeting of eight items with no exception at all 16.8 percent of the time. See two exceptions in eight items and the Wilson interval on the true rate runs from 7 to 59 percent — 52 points wide. One meeting cannot distinguish a well-run body from one taking half its decisions in the corridor.

So compare observed meetings with unobserved ones and measure the effect directly. Run the power calculation. With twelve meetings in each arm — a full year observed against a full year not — the minimum detectable difference in exception rate is 16.2 points. An audit of realistic size can detect an observer effect only if the observer effect is enormous.

Hold that beside the founding case. The Hawthorne illumination experiments gave the observer effect its name, and when Steven Levitt and John List went back to the original archival data in 2011 they found the canonical effect largely disappeared. The observer effect's own evidence did not survive an audit of the records.

Both findings point the same way, and it is the reverse of the intuition. Do not audit the room. Audit the archive. The archive is the only sample you will ever have that nobody performed for, it is five times larger than any field pass you can afford, it costs a fraction as much, and it was written before you arrived. Attend a meeting or two by all means — but attend to check that the minutes resemble the meeting, which is a validation sample, and say so.

Now the honest negative, and it is aimed at the instrument in your hands.

Cox and colleagues found the design principles supported. Jacopo Baggio and colleagues then examined sixty-nine cases configurationally and found something harder to live with: no single design principle is necessary and none is sufficient. They operate in combinations, and different combinations work in fisheries, forests, pastures and irrigation.

The consequence for anyone building a governance scorecard is severe and unwelcome. A score that sums eight principles is adding terms that do not add. Six of eight is not two-thirds of a functioning commons; it may be a working configuration or a broken one depending on which six. The same applies to the six measures in this chapter. They are a profile, not a total, and any software that reduces them to a single index has reintroduced exactly the fault II.10 spent a chapter on: composites communicate by hiding the weights, and a hidden weight will not be argued with.

And the second negative, which is the reason for the next movement. Three of these six measures are gameable by the people who must collect them. That is treated in Design, because the response is engineering, not disclosure.

DREAM

What becomes ordinary

In the organisation that has taken this seriously, the governance report is two pages and nobody dreads it.

The first page carries six numbers and their history. Effective participation, with the weights printed beside it the way a confidence interval is printed beside a mean. Excess latency, stated as days above the cycle floor, with the floor named. Reversal and non-implementation side by side, because everyone learned early that a body which never reverses anything is not decisive, it is unaccountable. The exception share, with its denominator stated in the same sentence. The effective number of voices in the minutes, beside the headcount. And the rule–practice divergence, in both directions: provisions that lie dormant, and practices with no written basis.

The second page is the boundary statement. What the audit looked at, what it did not, and how far the sample carries. It says, in plain numbers, that a year of one body's minutes supports a claim to about ten points and no finer, and that the finding above was therefore computed across four comparable bodies rather than four years of one.

The audit is run on the archive. Nobody sits in the back of the room with a clipboard, because the archive is larger, cheaper, and was written before the auditor existed. When somebody does attend, it is to check that the minutes resemble the meeting — a validation sample, declared as such.

The auditor is not paid by the body being audited, and this is unremarkable rather than adversarial; it is the same arrangement the accounts have had for a century. When a finding is uncomfortable, the response is to ask what the measure did not look at, not to ask who leaked it.

And the six numbers are attached to something. In one place it is a margin on a loan; in another a share of a surplus; in another simply the order of business, because a body that discovers half its provisions are dormant will spend an evening on them. A governance measure that is not attached to a decision is a pamphlet, and everybody in this imagined room learned that from the chapter on measurement rather than from a bad year.

The most ordinary thing about it is the tone. There is no dashboard, no colour coding, no score out of a hundred. Six numbers, their history, and the sentence saying what they missed.

DESIGN

The instrument, built — and the three measures that lie

Build in this order.

One: the decision register. Every decision the body takes gets one row: date of first external trace, date decided, route (standing process or exception), mover, outcome, and a field for the downstream artefact that proves it happened. Most bodies already have four of those six in their minutes. The register is usually an afternoon of extraction, not a new system.

Two: the archive pass, not the visit. Extract five years of minutes. The arithmetic is decisive. To estimate a proportion to ±5 points you need 385 decisions — but decisions inside one meeting are not independent draws, sharing a chair, an agenda and an evening, so at an intra-meeting correlation of 0.15 the design effect is 2.05 and you need 788. A body taking eight decisions at twelve meetings a year produces 96 a year: one year supports ±14.3 points, five years of archive ±6.4, and reaching ±5 points from a single body would take 8.2 years. Sample four comparable bodies for two years instead. The clustering is in the meeting, so sampling across bodies buys precision that sampling across time cannot.

Three: name the three gameable measures, and treat each differently. This is the movement's real content, and it is where an audit either becomes load-bearing or becomes theatre.

Participation is gamed by convening. Add meetings, mobilise a bloc, count everyone who signed in, count observers as participants. The auditor's answer is not to police the attendance sheet, which is a losing position, but to stop using it: compute effective participation from the decision record — who moved, who seconded, who was recorded in a minority — which is a document the body writes for a different purpose. Then keep one protected holdout in II.10's exact sense: the count of names appearing in the minutes who appear nowhere in any leadership role, never targeted, never in anybody's objectives. When attendance rises and that number does not, you have measured the gaming component directly.

Latency is gamed by starting the clock late. A body that logs a decision only once it is ripe reports superb latency and an invisible backlog. The auditor's answer is to date every decision from the first external trace — a member's written request, an email, a work order, a complaint — a record the body did not author for the audit. Then measure the un-started backlog as its own quantity, because a shrinking latency with a growing backlog is the signature of this particular game and it is unmistakable once both are on the page.

Reversal is gamed by never formally reversing. Nobody moves to rescind. The decision is allowed to lapse, re-scoped under a new name, or simply not implemented. The auditor's answer is to measure non-implementation instead: the share of decisions with no downstream artefact within twelve months. That number is computed from procurement records, work orders and payments — systems the governance body does not control — and it catches every variant of the game including the honest ones. Report silent reversal beside it: decisions revisited under a different title within the period, found by matching subject rather than wording.

Four: the rule–practice pass. Inventory every provision that creates a right or a duty and ask the archive whether it was exercised. In a worked example of 42 provisions, 11 had no trace in twelve months — a dormancy rate of 26.2 percent — while 6 standing practices had no written basis at all, a two-directional divergence of 35.4 percent. Neither document was written to be compared with the other, which is precisely why the comparison is hard to game, and why Jo Freeman's observation in "The Tyranny of Structurelessness" remains the sharpest thing ever written about the second direction: the rules that are not written down are the ones that cannot be contested.

Five: the minutes pass. Take a year of minutes and count recorded interventions by name. In the worked example, 24 names, 498 interventions, the top decile holding 33.9 percent, four people appearing exactly once, and an effective number of voices of 9.4 — 39.2 percent of the headcount. The headcount says twenty-four. The room has nine voices in it. That is the number the audit reports, and it is the one a chair can actually do something about.

DESTINY

How it holds when nobody is pushing

Three conditions, and the loss of any one ends it.

The auditor is not paid by the audited. The team computing the numbers cannot be the team judged on them. This is the structural feature most often dropped when the practice moves from a statistical office into an organisation, and it is the only one that cannot be replaced by good intentions.

The archive is protected before it is interesting. Minutes, registers and work orders get a retention rule written while nobody has a stake in them. An archive curated after a dispute is evidence of the dispute.

The holdout is defended by someone senior enough to refuse. The pressure to fold the untargeted count into somebody's objectives is constant, reasonable and eventually irresistible without one named person whose job is no.

Now the failure modes, plainly. It fails when the six numbers are summed into a score, because the Baggio finding says the terms do not add and because a score invites a target. It fails when a divergence between a targeted measure and its holdout is discovered and the response is to question the holdout — the most natural response available and the wrong one every time. It fails when the archive pass is run once, for a launch, and never again, at which point it is an artefact. And it fails when the audit is used to remove people rather than to change rules, after which the record quietly becomes less informative and stays that way for years, because everybody involved learned in one evening exactly what minutes are for.

DELIGHT

What it feels like

There is a specific pleasure in the records room, and it is the pleasure of the second reading: you are holding a document that somebody wrote carefully, for their own reasons, years before you arrived, with no idea that you would ever count anything in it.

It is absorbing in the way that sorting is absorbing. Five minute books, a pencil, a tally, and the slow accumulation of a shape nobody in the organisation has ever seen — not because it was hidden but because nobody had ever put five years on one table before.

And then the moment the tally turns into a sentence. The room has nine voices in it. Nobody had said that out loud. Everybody had felt it. You did not need to accuse anyone of anything, and no one has to defend themselves, because the number came from their own pages, written in their own hands, at a time when the question had not yet been asked.

That is the feeling, and it is the whole argument for auditing the record rather than the room: nobody was performing.

OPERATIONALIZE THIS

At the level of finance

Here is the instrument, in the form a treasurer and a lender both recognise.

The structure: a governance covenant with a margin ratchet, keyed to an archive audit.

This is a sustainability-linked loan with the sustainability performance targets replaced by governance measures. The mechanics are already standard under the Loan Market Association's Sustainability-Linked Loan Principles: an agreed set of targets, an independent verifier, and a margin that steps down when the targets are met and up when they are not. Nothing has to be invented. The novelty is only in what is measured, and governance is the easiest thing a cooperative or a community body has to offer a lender, because it is the thing they already do best and never get paid for.

The mechanics.

Targets. Three of the six, chosen for the borrower's actual risk: normally effective participation, non-implementation rate, and rule–practice divergence. Never a composite. Each reported with its boundary statement.
Computation. From the archive, by a verifier engaged and paid by the lender, not the borrower. Five years of minutes at the first pass, then rolling.
Holdout. One measure computed and published every period and excluded from the ratchet by written term, so both parties can see the gaming component.
The step. Twenty-five basis points on a five-million-pound facility is £12,500 a year. A fully loaded archive audit and independent verification runs about £11,500, giving a cover ratio of 1.09× — it clears, and it clears narrowly, which is the honest way to present it.
Breakeven. 23.0 basis points at that facility size, or a facility of £4.6 million at a twenty-five point step. Below either, the covenant costs more than it saves and should be run as a plain annual audit with no ratchet.
Term and review. Three years, with the target levels re-set once on evidence rather than annually on negotiation. A target renegotiated every year is a target nobody has to meet.

The balance-sheet treatment. The audit fee is operating expenditure. The margin saving arrives in interest expense, so the whole instrument shows up in two lines a finance committee already reads. There is no new asset, no capitalisation question and no auditor conversation — which is precisely why it can be done this year rather than after a policy review.

The counterparty. A cooperative or mutual lender first, or a community development finance institution, because they already price governance informally and will recognise the proposition immediately. A mainstream lender second, once you hold two completed periods, at which point you are presenting a verified series rather than a proposal.

The number that decides it.

        annual margin saving
   ---------------------------------  >  1.0
    audit cost + verification cost

If that holds, the governance audit is free and the covenant is paying for it. If it does not, run the audit anyway and drop the ratchet: the six measures earn their keep at £11,500 whether or not a lender is watching, and the first pass is the one that finds the eleven dormant provisions.

The first ninety days.

Day	Action	Artifact
1–15	Pull five years of minutes; build the decision register	The register
16–30	Code route, mover, outcome, first external trace	Coded register
31–45	Compute the six; publish weights and boundary	The two-page report
46–60	Inventory provisions; run the rule–practice pass	Dormancy list
61–75	Name the holdout; get it excluded in writing	The holdout term
76–90	Take the covenant to one lender	Term sheet

APPRECIATIVE QUESTIONS

Twelve, for a room

Discovery — what is already working

Think of a decision this body took in the last two years that still looks right. Walk us through how it actually got made — who first raised it, and what happened between then and the vote?
Which of our records do we keep carefully for our own reasons, with nobody requiring it? What made us start?
When has somebody in this room changed their mind in a meeting and said so out loud? What made that possible that day?

Dream — what becomes possible

If our governance report were two pages and we looked forward to it, what would be on the first page?
Imagine the minutes of this body five years from now being read by somebody who was not here. What would we want them to be able to work out?
If every member could see exactly how a decision reaches them, what would change about who comes?

Design — what we build

What is the one record we already produce that would tell us most about how we actually decide, if somebody simply counted it?
Which measure would we most want to be judged on — and who should compute it, given that it should not be us?
What would we choose as our holdout: the number we agree now that nobody may ever target?

Destiny — how it holds

What would have to be true for this audit to still be running when everyone currently in this room has moved on?
Which of our written rules have we not used in a year, and what would it take to either use them or retire them honestly?
What is the first sign we would see that our own minutes had started telling us less than they used to, and who would notice it first?

WORKS CITED

Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge University Press.

Cox, M., Arnold, G. and Villamayor-Tomás, S. (2010). "A Review of Design Principles for Community-based Natural Resource Management." Ecology and Society, 15(4), 38.

Baggio, J. A., Barnett, A. J., Perez-Ibarra, I., Brady, U., Ratajczyk, E., Rollins, N., Rubiños, C., Shin, H. C., Yu, D. J., Aggarwal, R., Anderies, J. M. and Janssen, M. A. (2016). "Explaining success and failure in the commons: the configural nature of Ostrom's institutional design principles." International Journal of the Commons, 10(2), 417–439.

Chhatre, A. and Agrawal, A. (2008). "Forest commons and local enforcement." Proceedings of the National Academy of Sciences, 105(36), 13186–13191.

Persha, L., Agrawal, A. and Chhatre, A. (2011). "Social and Ecological Synergy: Local Rulemaking, Forest Livelihoods, and Biodiversity Conservation." Science, 331(6024), 1606–1608.

Baiocchi, G. (2005). Militants and Citizens: The Politics of Participatory Democracy in Porto Alegre. Stanford University Press.

Wampler, B. (2007). Participatory Budgeting in Brazil: Contestation, Cooperation, and Accountability. Pennsylvania State University Press.

Avritzer, L. (2009). Participatory Institutions in Democratic Brazil. Johns Hopkins University Press.

Cabannes, Y. (2004). "Participatory budgeting: a significant contribution to participatory democracy." Environment and Urbanization, 16(1), 27–46.

Gonçalves, S. (2014). "The Effects of Participatory Budgeting on Municipal Expenditures and Infant Mortality in Brazil." World Development, 53, 94–110.

Touchton, M. and Wampler, B. (2014). "Improving Social Well-Being Through New Democratic Institutions." Comparative Political Studies, 47(10), 1442–1469.

Myners, P. (2014). The Co-operative Group: Report of the Independent Governance Review. The Co-operative Group.

Laakso, M. and Taagepera, R. (1979). "'Effective' Number of Parties: A Measure with Application to West Europe." Comparative Political Studies, 12(1), 3–27.

Simpson, E. H. (1949). "Measurement of Diversity." Nature, 163, 688.

Wilson, E. B. (1927). "Probable Inference, the Law of Succession, and Statistical Inference." Journal of the American Statistical Association, 22(158), 209–212.

Kish, L. (1965). Survey Sampling. Wiley.

Campbell, D. T. (1979). "Assessing the Impact of Planned Social Change." Evaluation and Program Planning, 2(1), 67–90.

Goodhart, C. A. E. (1984). "Problems of Monetary Management: The U.K. Experience," in Monetary Theory and Practice: The U.K. Experience. Macmillan.

Strathern, M. (1997). "'Improving ratings': audit in the British University system." European Review, 5(3), 305–321.

Power, M. (1997). The Audit Society: Rituals of Verification. Oxford University Press.

Bevan, G. and Hood, C. (2006). "What's Measured Is What Matters: Targets and Gaming in the English Public Health Care System." Public Administration, 84(3), 517–538.

Roethlisberger, F. J. and Dickson, W. J. (1939). Management and the Worker. Harvard University Press.

Levitt, S. D. and List, J. A. (2011). "Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments." American Economic Journal: Applied Economics, 3(1), 224–238.

Freeman, J. (1972). "The Tyranny of Structurelessness." Berkeley Journal of Sociology, 17, 151–164.

Whyte, W. F. and Whyte, K. K. (1991). Making Mondragón: The Growth and Dynamics of the Worker Cooperative Complex, 2nd edn. ILR Press.

Loan Market Association (2023). Sustainability-Linked Loan Principles. LMA, APLMA and LSTA.

Note on figures. Effective participation, latency, sampling precision, design effect, minimum detectable effect, concentration, divergence and the covenant arithmetic are all computed in lib/verify/VI_10.py and reproducible there, with every input printed and every assumed parameter labelled. Porto Alegre population is the 2000 IBGE census; the annual participation figure is the published order of magnitude and is used as such. Figures marked ILLUSTRATIVE in that module are worked examples demonstrating a computation, not measurements of a named body, and the module says so on its face.