Management dashboards that support defensible decisions IDER in-house training materials Hypothetical cases and figures for teaching the methods. Learning objectives Define reproducible indicators with an explicit observation grain and denominator. Diagnose composition reversals and calculate standardised rates. Connect uncertainty, data quality and action ownership in a dashboard. Basic percentages and experience using tabular operational data. Build a dashboard that improves decisions Decision before display Denominator before colour All numerical cases are hypothetical A dashboard reports that service success increased sharply, yet both district managers say performance deteriorated. Your task is to determine whether the display is wrong, the managers are wrong, or both are describing different quantities. This clinic uses a hypothetical two-district service programme to teach metric design, aggregation, uncertainty and operational response. We will move beyond attractive charts to an auditable chain from records to decisions. Keep a calculator nearby and sketch the denominator for every rate. Junior practitioners should reproduce the calculations; senior practitioners should examine whether each indicator answers the intended management question. By the end, you should be able to build a metric contract, detect a composition reversal and specify actions that follow from evidence rather than colour alone. Give every indicator a decision and an owner What decision can change? Who acts, and by when? What evidence would trigger action? Begin with a management decision that can actually be changed. A district supervisor may reassign caseworkers next week, while an executive may change quarterly staffing. Those decisions need different time horizons and levels of detail. Write down the action owner, review frequency and available response before selecting a metric. A red indicator without an owner is a warning with no operating mechanism. Also define the evidence needed to distinguish a capacity shortage from incomplete reporting. For example, falling resolution rates alongside a growing backlog and stable case mix suggest a different response from a sudden loss of uploads. The dashboard should make the next investigation clear, not imply that every deterioration has the same cause or deserves the same intervention. Define grain, numerator and denominator together One row: one eligible case episode Success within a fixed follow-up window Include eligible unresolved cases in the denominator Suppose the indicator is the share of eligible cases resolved within thirty days. Define a case episode so repeat contacts do not become extra successes. Record an eligibility date, a unique identifier and the resolution date. The denominator should include eligible episodes with enough follow-up, including those still unresolved. If you divide by completed cases only, the hardest unresolved cases disappear and the rate looks artificially favourable. Write exclusions explicitly, such as test records or genuine duplicates, and apply them consistently across districts and periods. Store a metric contract containing the definition, grain, date logic, responsible owner and version. Another analyst should reproduce the same result from the same frozen data extract without needing verbal instructions from the original dashboard developer. Test the route from source records to a rate Validate unique episode identifiers Check joins and reporting completeness Reconcile counts before publishing Before calculating a rate, trace how each source contributes rows. Joining one case to several contact records can multiply it, inflating both totals and apparent activity. Test uniqueness at the intended grain before and after every join. Reconcile district counts with source control totals and quantify records excluded by date, eligibility or missing fields. Then test freshness separately from completeness: yesterday's extraction can still contain only half the expected district returns. Retain a snapshot identifier and calculation timestamp so a previous displayed result can be reproduced. These checks are operational controls, not cosmetic annotations. When a rate changes unexpectedly, they help establish whether the change reflects service performance, a changed population, delayed reporting or a transformation error in the analytical pipeline. A pooled rate weights districts by eligible cases Period 1: A = 90/100; B = 90/900 Pooled success = 180/1,000 = 18% Average district rate = 50% In period one, district A resolves ninety of one hundred eligible cases, while district B resolves ninety of nine hundred. Their rates are ninety percent and ten percent. The pooled case success rate is one hundred and eighty divided by one thousand, or eighteen percent. Averaging ninety and ten gives fifty percent, but that answers a different question: the mean of two equally weighted district rates. Neither quantity should be chosen because it looks better. If management asks about an eligible person's overall chance of success in these records, use the pooled rate. If it asks about the average district with equal district importance, label that estimand explicitly. Always carry denominators with rates so the aggregation logic remains visible and auditable. A rising total can conceal declining district rates Period 2: A = 720/900 = 80% Period 2: B = 5/100 = 5% Pooled rate rises to 72.5% Now district A resolves seven hundred and twenty of nine hundred eligible cases, while district B resolves five of one hundred. A declines from ninety to eighty percent, and B declines from ten to five percent. Yet the pooled rate becomes seven hundred and twenty-five divided by one thousand, or seventy-two point five percent. More of the second period's cases come from the higher-performing district. This composition shift outweighs the declines within both districts. The aggregate calculation is arithmetically correct, but calling it an improvement in district performance would mislead. Show district results alongside the pooled series and investigate why the case mix changed. The operational interpretation depends on whether those shifts reflect demand, reassignment, eligibility changes or incomplete reporting. Hold the district mix constant Period 1 shares: A 10%; B 90% Period 2 rates: A 80%; B 5% Calculate the standardised period 2 rate Pause and calculate what the second period's success rate would be if district shares remained at their first-period levels. Multiply district A's new rate by ten percent and district B's new rate by ninety percent, then add the contributions. Compare that standardised result with the first period's pooled eighteen percent. Write one sentence explaining why the standardised rate differs from the observed seventy-two point five percent. Then identify a management question answered by each number. Avoid labelling one number true and the other false: they describe different population compositions. Your answer should make the reference weights explicit so a colleague can reproduce it and understand why a different reference population could produce a different standardised result. Standardisation reveals a 5.5-point deterioration 0.10 × 80% + 0.90 × 5% = 12.5% 12.5% − 18% = −5.5 percentage points Observed and standardised rates answer different questions 0.10 × 80% + 0.90 × 5% = 12.5% Ten percent of eighty percent contributes eight percentage points. Ninety percent of five percent contributes four point five points. Adding them gives a standardised rate of twelve point five percent, which is five point five points below the first period's eighteen percent. The observed seventy-two point five percent describes outcomes for the actual second-period mix. The standardised result describes how second-period district rates would combine under the chosen first-period mix. This adjustment helps compare performance while holding one compositional dimension constant. It does not automatically adjust for differences within districts, such as case severity or referral source. Publish the reference weights, keep the unadjusted series available and explain why this particular standard population fits the management question being investigated. Adjust only for variables justified by the question Separate case mix from programme mechanisms Check overlap before comparing units Preserve raw and adjusted results Case-mix adjustment can improve comparison, but indiscriminate adjustment can also hide meaningful performance. Suppose staff support changes how quickly cases become complex. Adjusting for complexity measured after support begins may remove part of the programme pathway. Prefer baseline characteristics when the aim is to compare like with like, and explain the causal role of each adjustment variable. Check overlap: comparing districts with entirely different case profiles may require extrapolation rather than direct evidence. Report raw results alongside adjusted estimates, with the model or weighting rule documented. Do not present an adjusted ranking as a league table of managerial quality unless the design supports that interpretation. Adjustment addresses specified differences; it cannot guarantee that all relevant differences have been measured or handled correctly. Choose uncertainty measures that match the data source Survey sample: quantify sampling uncertainty Administrative census: inspect coverage and measurement Targets do not remove uncertainty A rate from a probability sample has sampling uncertainty because different selected people could produce a different estimate. A complete administrative count of a defined population does not acquire sampling uncertainty merely because it appears on a dashboard. It still faces measurement error, missing records, duplicate episodes and delayed updates. Distinguish these sources rather than attaching a generic confidence interval to every number. For a sampled indicator, use weights and design information when appropriate. For an administrative indicator, show reporting coverage, unresolved validation issues and revision status. A target line is a management benchmark, not an uncertainty calculation. When the displayed value sits close to that line, the evidence quality and consequences of a wrong response should influence the action taken. Compute the sampling margin before choosing a status Sample success: 60%; simple random n = 400 Standard error ≈ 2.45 percentage points 95% interval ≈ 55.20% to 64.80% Assume a simple random sample of four hundred independent observations from a much larger population, with sixty percent successful. The standard error of a proportion is approximately the square root of the proportion times one minus the proportion, divided by sample size. Here, the square root of zero point six times zero point four divided by four hundred is about zero point zero two four five. Multiply by one point nine six to obtain a margin of roughly four point eight percentage points. The approximate interval runs from fifty-five point two to sixty-four point eight percent. This normal approximation is reasonable for these illustrative counts, but actual survey designs may require different standard errors. Keep that design assumption next to the calculation. Separate a signal, an investigation and a response Signal: evidence crosses a defined rule Investigation: identify the likely mechanism Response: owner, deadline and follow-up A dashboard should not jump directly from a changed number to a disciplinary or spending decision. Define a signal rule, then an investigation step and an authorised response. For example, a sustained backlog increase might trigger a supervisor review of staffing, case complexity and reporting completeness. The review determines whether temporary staff, process repair or data correction is appropriate. Record the action, owner and review date in the same operating system used for follow-up. Also consider false alarms when monitoring many indicators repeatedly; isolated threshold crossings can occur by chance. Use a documented monitoring method suitable for the process and its dependence over time. The practical aim is a reliable feedback loop in which evidence changes action and subsequent data show whether that action helped. Can 60% be confidently labelled above a 58% target? Estimate: 60%; simple random n = 400 Target: 58% Then consider a design effect of 2 Pause and decide whether the sample result supports an automatic green status meaning performance is established above the target. Use the interval we just calculated, not the point estimate alone. Then consider a survey design effect of two. Multiply the original variance by two, which means multiplying the standard error by the square root of two. Recalculate the approximate ninety-five percent interval and explain what changed. The design effect is supplied for this exercise; in practice it must correspond to the outcome and estimator. Finally, propose a useful dashboard label that distinguishes the observed estimate from the strength of evidence about the target. The goal is a clear management message, not a display that suggests unwarranted certainty through colour. The target lies inside both uncertainty intervals Simple random interval: 55.20%–64.80% Design effect 2 interval: 53.21%–66.79% Point estimate above target; conclusion uncertain The simple random interval includes fifty-eight percent, so the result does not establish that underlying performance exceeds the target. With a design effect of two, the standard error rises to about three point four six percentage points. Multiplying by one point nine six gives a margin of about six point seven nine points, producing an interval from fifty-three point two one to sixty-six point seven nine percent. The target remains inside, with greater uncertainty. A useful label is: estimate above target, evidence not conclusive. Management may still act based on risk and cost, but should not confuse that decision with statistical proof. If repeated monitoring is intended, define a consistent monitoring approach rather than treating every new interval as an independent test. Make limitations visible where decisions are made Show denominator, period and freshness Use accessible labels and comparable scales Link the displayed number to its definition Place the information needed for interpretation next to the indicator: numerator, denominator, period, target definition and data status. Use direct labels and text as well as colour so the display remains understandable to people with colour-vision differences and on monochrome exports. Keep time scales and axes consistent where comparisons depend on them, and avoid truncating a bar axis in a way that exaggerates changes. Provide a route from the summary to district detail and the metric definition without exposing personal records. For revised data, show that a revision occurred and retain a reproducible snapshot. A visually restrained dashboard can carry substantial analytical depth when the hierarchy makes the decision, the evidence and the uncertainty easy to find in the right order. Deliver a metric system with an operating routine Metric contract and reproducible calculations Composition and uncertainty checks Action log with ownership and review dates Choose one indicator in your own organisation and rebuild it as a complete decision process. Write its metric contract, trace the source records and reproduce its numerator and denominator. Test whether a change in population composition explains the recent trend. Identify the appropriate uncertainty or data-quality statement and agree an action rule with the person responsible for responding. Then run a short review meeting using the dashboard: ask what changed, whether the evidence is reliable, what mechanism is plausible and what action will be checked next time. Keep the resulting decision log. Your deliverable is more than a screen. It is a repeatable connection between well-defined measurements, transparent interpretation and accountable action that can withstand questions from both analysts and operational managers. Check your understanding 1. A resolves 90/100 cases and B resolves 90/900. What is the pooled case success rate? A. 50% B. 18% C. 90% D. 10% 2. Both district rates fall while the pooled rate rises. What should you investigate first? A. Whether the population shares changed B. Whether arithmetic can be ignored C. Whether every district improved D. Whether the target should be removed 3. Period 1 district shares are 10% and 90%. Period 2 rates are 80% and 5%. What is the standardised rate? A. 72.5% B. 42.5% C. 12.5% D. 8% 4. A 60% sample estimate has a 95% interval of 55.20%–64.80%. The target is 58%. Which label is justified? A. Target exceeded with certainty B. Estimate above target; evidence not conclusive C. Target definitely missed D. No uncertainty because the estimate exceeds 58% 5. What happens to standard error when design effect is 2? A. It doubles B. It is divided by 2 C. It is multiplied by √2 D. It stays unchanged 6. A complete administrative census has missing or duplicate records. What is most appropriate? A. Add a generic sampling interval to solve the problem B. Report coverage and measurement limitations and repair the records C. Treat all errors as random sampling D. Hide the denominator Applied task 1 Create a metric contract for thirty-day case resolution, including date logic, duplicates, unresolved cases, owner and action. Applied task 2 Explain the two-period district example to an executive using observed and standardised rates, then propose one investigation. Answers and explanations 1. B. Pooling uses 180 successes divided by 1,000 eligible cases. Fifty percent is an equally weighted average of district rates. (Slide 5) 2. A. A shift toward the higher-rate district can raise the total despite deterioration within each district. (Slide 6) 3. C. 0.10 × 80% + 0.90 × 5% = 8% + 4.5% = 12.5%. (Slide 8) 4. B. The interval contains the target, so an above-target point estimate does not establish an above-target population value. (Slide 14) 5. C. Design effect multiplies variance; standard error is the square root of variance. (Slide 14) 6. B. Sampling intervals do not remove administrative coverage, duplication or measurement errors. (Slide 10) Applied task 1 Use one eligible case episode as the grain. Include episodes with a full thirty-day observation window in the denominator, count resolutions within that window in the numerator, retain unresolved eligible cases, and apply documented duplicate rules. State snapshot, refresh schedule, owner and an investigation before operational action. Numerator and denominator share an explicit grain and window. Missingness, duplicates and reporting completeness have checks. A named role owns a defined review and follow-up action. Applied task 2 The observed rate rises from 18% to 72.5% as cases shift toward A. Both local rates fall. Holding initial shares fixed gives 12.5%, a 5.5-point deterioration. Investigate whether the composition shift reflects demand, reassignment, eligibility or missing district reporting before attributing it to management. Correct pooled and standardised calculations. Clearly distinguishes performance change from composition. Proposes an evidence-based investigation without a causal overclaim. References ONS — Uncertainty and how we measure it for our surveys https://www.ons.gov.uk/methodology/methodologytopicsandstatisticalconcepts/uncertaintyandhowwemeasureit Sampling uncertainty, confidence intervals and non-sampling limitations. J-PAL — Introduction to measurement and indicators https://www.povertyactionlab.org/resource/introduction-measurement-and-indicators Indicator definitions and measurement choices. J-PAL — Data visualization https://www.povertyactionlab.org/resource/data-visualization Clear quantitative communication and interpretable displays.