What you will be able to do
- Define reproducible indicators with an explicit observation grain and denominator.
- Diagnose composition reversals and calculate standardised rates.
- Connect uncertainty, data quality and action ownership in a dashboard.
Before you begin: Basic percentages and experience using tabular operational data.
Give every indicator a decision and an owner
Begin with a management decision that can actually be changed. A district supervisor may reassign caseworkers next week, while an executive may change quarterly staffing. Those decisions need different time horizons and levels of detail. Write down the action owner, review frequency and available response before selecting a metric. A red indicator without an owner is a warning with no operating mechanism. Also define the evidence needed to distinguish a capacity shortage from incomplete reporting. For example, falling resolution rates alongside a growing backlog and stable case mix suggest a different response from a sudden loss of uploads. The dashboard should make the next investigation clear, not imply that every deterioration has the same cause or deserves the same intervention.
A pooled rate weights districts by eligible cases
In period one, district A resolves ninety of one hundred eligible cases, while district B resolves ninety of nine hundred. Their rates are ninety percent and ten percent. The pooled case success rate is one hundred and eighty divided by one thousand, or eighteen percent. Averaging ninety and ten gives fifty percent, but that answers a different question: the mean of two equally weighted district rates. Neither quantity should be chosen because it looks better. If management asks about an eligible person's overall chance of success in these records, use the pooled rate. If it asks about the average district with equal district importance, label that estimand explicitly. Always carry denominators with rates so the aggregation logic remains visible and auditable.
Choose uncertainty measures that match the data source
A rate from a probability sample has sampling uncertainty because different selected people could produce a different estimate. A complete administrative count of a defined population does not acquire sampling uncertainty merely because it appears on a dashboard. It still faces measurement error, missing records, duplicate episodes and delayed updates. Distinguish these sources rather than attaching a generic confidence interval to every number. For a sampled indicator, use weights and design information when appropriate. For an administrative indicator, show reporting coverage, unresolved validation issues and revision status. A target line is a management benchmark, not an uncertainty calculation. When the displayed value sits close to that line, the evidence quality and consequences of a wrong response should influence the action taken.
Make limitations visible where decisions are made
Place the information needed for interpretation next to the indicator: numerator, denominator, period, target definition and data status. Use direct labels and text as well as colour so the display remains understandable to people with colour-vision differences and on monochrome exports. Keep time scales and axes consistent where comparisons depend on them, and avoid truncating a bar axis in a way that exaggerates changes. Provide a route from the summary to district detail and the metric definition without exposing personal records. For revised data, show that a revision occurred and retain a reproducible snapshot. A visually restrained dashboard can carry substantial analytical depth when the hierarchy makes the decision, the evidence and the uncertainty easy to find in the right order.
Apply the method
Work through each task before opening the model answer. Compare your reasoning with the review criteria. Written answers stay on this page and clear when you leave or reload it.
Applied task 1
Show model answer and review criteria
Use one eligible case episode as the grain. Include episodes with a full thirty-day observation window in the denominator, count resolutions within that window in the numerator, retain unresolved eligible cases, and apply documented duplicate rules. State snapshot, refresh schedule, owner and an investigation before operational action.
- Numerator and denominator share an explicit grain and window.
- Missingness, duplicates and reporting completeness have checks.
- A named role owns a defined review and follow-up action.
Applied task 2
Show model answer and review criteria
The observed rate rises from 18% to 72.5% as cases shift toward A. Both local rates fall. Holding initial shares fixed gives 12.5%, a 5.5-point deterioration. Investigate whether the composition shift reflects demand, reassignment, eligibility or missing district reporting before attributing it to management.
- Correct pooled and standardised calculations.
- Clearly distinguishes performance change from composition.
- Proposes an evidence-based investigation without a causal overclaim.
Method references
- ONS — Uncertainty and how we measure it for our surveys
Sampling uncertainty, confidence intervals and non-sampling limitations.
- J-PAL — Introduction to measurement and indicators
Indicator definitions and measurement choices.
- J-PAL — Data visualization
Clear quantitative communication and interpretable displays.