A protocol for measuring reviewer disagreement in bookkeeping

Select one recurring workpaper type and one defined classification task, such as assigning an approved exception category. Freeze the eligible population before drawing the sample. Each observation is one item independently classified by two reviewers using the same evidence snapshot and label guide.

The guide should define every label, boundaries between labels, and the treatment of insufficient information. GAO's content-analysis guide provides the methodological basis for explicit categories, coder training, pilot testing, and documented decisions. Applying that approach to bookkeeping exceptions is this brief's proposed adaptation.

Freeze sample IDs, evidence versions, and definitions before production coding. Reviewers work independently and cannot see each other's choice until both entries are locked. Record each initial label, an abstention or insufficient-evidence code where allowed, and any evidence unavailable to both reviewers.

Training examples do not belong in the measured sample. Items reviewed by only one person are coverage exceptions rather than agreements. If a reviewer knows material context outside the shared packet, record that condition and resolve the evidence design before interpreting the label pair.

The raw disagreement rate is the number of eligible dual-coded items with different initial labels divided by all eligible dual-coded items. Apply the preregistered rule for abstentions and publish the label-by-label count table. A table reveals whether differences cluster around one boundary that a single percentage conceals.

Keep the adjudicated label separate from the locked initial pair. If a chance-adjusted measure is used, specify the statistic in advance and report sample size and category prevalence beside it. GAO-13-591SP discusses both agreement assessment and the limitations of simple and chance-corrected summaries; it does not make a particular statistic suitable for every bookkeeping sample.

After calculation, an authorized adjudicator can examine disagreements against the shared evidence and approved rule. The outcome may affirm one label, identify insufficient evidence, or require a prospective definition change. Do not overwrite the original entries, and do not back-apply a revised guide without labeling a new analysis version.

The method owner controls sampling and calculation. Reviewers classify; the adjudicator resolves; the finance owner retains authority over accounting treatment. GAO's internal-control guidance supports assigned responsibility and documented evaluation, but it supplies no pass rate for this exercise. Results should be used for calibration, not as a stand-alone performance score.

Observed disagreement depends on label prevalence, sample difficulty, evidence completeness, prior calibration, and reviewer familiarity. A sample deliberately enriched for ambiguous cases cannot be compared fairly with a routine random sample. High raw agreement can also occur when nearly every item receives one dominant label.

Agreement does not prove correctness, and adjudication can involve judgment. Small or sparse samples produce unstable category rates, and results do not automatically transfer to another workpaper or reviewer pair. A report should disclose selection, label definitions, category counts, abstentions, evidence gaps, outside context, adjudication, and guide changes.