WEBVTT NOTE IDER in-house training materials. Captions follow measured phrase audio timings. Hypothetical worked cases; figures are instructional examples. 00:00:00.000 --> 00:00:06.913 A dashboard reports that service success increased sharply, yet both district managers say performance 00:00:06.913 --> 00:00:07.407 deteriorated. 00:00:07.527 --> 00:00:14.925 Your task is to determine whether the display is wrong, the managers are wrong, or both are describing 00:00:14.925 --> 00:00:15.747 different quantities. 00:00:15.867 --> 00:00:21.856 This clinic uses a hypothetical two-district service programme to teach metric design, aggregation, 00:00:21.856 --> 00:00:28.765 uncertainty and operational response. We will move beyond attractive charts to an auditable chain from 00:00:28.765 --> 00:00:30.147 records to decisions. 00:00:30.267 --> 00:00:37.720 Keep a calculator nearby and sketch the denominator for every rate. Junior practitioners should reproduce 00:00:37.720 --> 00:00:44.178 the calculations; senior practitioners should examine whether each indicator answers the intended management 00:00:44.178 --> 00:00:44.675 question. 00:00:44.795 --> 00:00:51.754 By the end, you should be able to build a metric contract, detect a composition reversal and specify 00:00:51.754 --> 00:00:55.233 actions that follow from evidence rather than colour alone. 00:00:55.903 --> 00:01:02.283 Begin with a management decision that can actually be changed. A district supervisor may reassign 00:01:02.283 --> 00:01:06.537 caseworkers next week, while an executive may change quarterly staffing. 00:01:06.657 --> 00:01:13.554 Those decisions need different time horizons and levels of detail. Write down the action owner, review 00:01:13.554 --> 00:01:17.002 frequency and available response before selecting a metric. 00:01:17.122 --> 00:01:23.954 A red indicator without an owner is a warning with no operating mechanism. Also define the evidence needed 00:01:23.954 --> 00:01:26.990 to distinguish a capacity shortage from incomplete reporting. 00:01:27.110 --> 00:01:33.546 For example, falling resolution rates alongside a growing backlog and stable case mix suggest a 00:01:33.546 --> 00:01:36.979 different response from a sudden loss of uploads. 00:01:37.099 --> 00:01:43.567 The dashboard should make the next investigation clear, not imply that every deterioration has the same 00:01:43.567 --> 00:01:45.992 cause or deserves the same intervention. 00:01:46.662 --> 00:01:54.595 Suppose the indicator is the share of eligible cases resolved within thirty days. Define a case episode so 00:01:54.595 --> 00:01:57.680 repeat contacts do not become extra successes. 00:01:57.800 --> 00:02:05.136 Record an eligibility date, a unique identifier and the resolution date. The denominator should include 00:02:05.136 --> 00:02:09.538 eligible episodes with enough follow-up, including those still unresolved. 00:02:09.658 --> 00:02:16.923 If you divide by completed cases only, the hardest unresolved cases disappear and the rate looks 00:02:16.923 --> 00:02:17.831 artificially favourable. 00:02:17.951 --> 00:02:25.309 Write exclusions explicitly, such as test records or genuine duplicates, and apply them consistently across 00:02:25.309 --> 00:02:31.686 districts and periods. Store a metric contract containing the definition, grain, date logic, 00:02:31.686 --> 00:02:33.648 responsible owner and version. 00:02:33.768 --> 00:02:40.303 Another analyst should reproduce the same result from the same frozen data extract without needing verbal 00:02:40.303 --> 00:02:42.754 instructions from the original dashboard developer. 00:02:43.424 --> 00:02:50.676 Before calculating a rate, trace how each source contributes rows. Joining one case to several contact 00:02:50.676 --> 00:02:55.208 records can multiply it, inflating both totals and apparent activity. 00:02:55.328 --> 00:03:02.334 Test uniqueness at the intended grain before and after every join. Reconcile district counts with source 00:03:02.334 --> 00:03:07.588 control totals and quantify records excluded by date, eligibility or missing fields. 00:03:07.708 --> 00:03:14.616 Then test freshness separately from completeness: yesterday's extraction can still contain only half the 00:03:14.616 --> 00:03:21.031 expected district returns. Retain a snapshot identifier and calculation timestamp so a previous 00:03:21.031 --> 00:03:23.498 displayed result can be reproduced. 00:03:23.618 --> 00:03:28.517 These checks are operational controls, not cosmetic annotations. 00:03:28.637 --> 00:03:36.512 When a rate changes unexpectedly, they help establish whether the change reflects service performance, a 00:03:36.512 --> 00:03:42.813 changed population, delayed reporting or a transformation error in the analytical pipeline. 00:03:43.483 --> 00:03:49.858 In period one, district A resolves ninety of one hundred eligible cases, while district B resolves 00:03:49.858 --> 00:03:54.640 ninety of nine hundred. Their rates are ninety percent and ten percent. 00:03:54.760 --> 00:04:00.797 The pooled case success rate is one hundred and eighty divided by one thousand, or eighteen percent. 00:04:00.917 --> 00:04:08.036 Averaging ninety and ten gives fifty percent, but that answers a different question: the mean of two equally 00:04:08.036 --> 00:04:12.783 weighted district rates. Neither quantity should be chosen because it looks better. 00:04:12.903 --> 00:04:19.032 If management asks about an eligible person's overall chance of success in these records, use the pooled 00:04:19.032 --> 00:04:19.393 rate. 00:04:19.513 --> 00:04:25.844 If it asks about the average district with equal district importance, label that estimand explicitly. 00:04:25.844 --> 00:04:31.332 Always carry denominators with rates so the aggregation logic remains visible and auditable. 00:04:32.001 --> 00:04:38.477 Now district A resolves seven hundred and twenty of nine hundred eligible cases, while district B resolves 00:04:38.477 --> 00:04:40.001 five of one hundred. 00:04:40.121 --> 00:04:45.090 A declines from ninety to eighty percent, and B declines from ten to five percent. 00:04:45.210 --> 00:04:51.731 Yet the pooled rate becomes seven hundred and twenty-five divided by one thousand, or seventy-two 00:04:51.731 --> 00:04:57.818 point five percent. More of the second period's cases come from the higher-performing district. 00:04:57.938 --> 00:05:03.939 This composition shift outweighs the declines within both districts. The aggregate calculation is 00:05:03.939 --> 00:05:09.479 arithmetically correct, but calling it an improvement in district performance would mislead. 00:05:09.599 --> 00:05:18.125 Show district results alongside the pooled series and investigate why the case mix changed. The operational 00:05:18.125 --> 00:05:24.520 interpretation depends on whether those shifts reflect demand, reassignment, eligibility changes or 00:05:24.520 --> 00:05:25.586 incomplete reporting. 00:05:26.256 --> 00:05:32.935 Pause and calculate what the second period's success rate would be if district shares remained at their 00:05:32.935 --> 00:05:33.721 first-period levels. 00:05:33.841 --> 00:05:41.348 Multiply district A's new rate by ten percent and district B's new rate by ninety percent, then add the 00:05:41.348 --> 00:05:46.089 contributions. Compare that standardised result with the first period's pooled eighteen percent. 00:05:46.209 --> 00:05:52.507 Write one sentence explaining why the standardised rate differs from the observed seventy-two point five 00:05:52.507 --> 00:05:56.705 percent. Then identify a management question answered by each number. 00:05:56.825 --> 00:06:03.048 Avoid labelling one number true and the other false: they describe different population compositions. 00:06:03.168 --> 00:06:09.847 Your answer should make the reference weights explicit so a colleague can reproduce it and understand why a 00:06:09.847 --> 00:06:13.187 different reference population could produce a different standardised result. 00:06:13.857 --> 00:06:19.862 Ten percent of eighty percent contributes eight percentage points. Ninety percent of five percent 00:06:19.862 --> 00:06:22.007 contributes four point five points. 00:06:22.127 --> 00:06:28.960 Adding them gives a standardised rate of twelve point five percent, which is five point five points below 00:06:28.960 --> 00:06:30.858 the first period's eighteen percent. 00:06:30.978 --> 00:06:37.926 The observed seventy-two point five percent describes outcomes for the actual second-period mix. The 00:06:37.926 --> 00:06:43.882 standardised result describes how second-period district rates would combine under the chosen 00:06:43.882 --> 00:06:44.875 first-period mix. 00:06:44.995 --> 00:06:51.111 This adjustment helps compare performance while holding one compositional dimension constant. It does 00:06:51.111 --> 00:06:57.696 not automatically adjust for differences within districts, such as case severity or referral source. 00:06:57.816 --> 00:07:04.909 Publish the reference weights, keep the unadjusted series available and explain why this particular 00:07:04.909 --> 00:07:08.962 standard population fits the management question being investigated. 00:07:09.632 --> 00:07:15.616 Case-mix adjustment can improve comparison, but indiscriminate adjustment can also hide meaningful 00:07:15.616 --> 00:07:20.603 performance. Suppose staff support changes how quickly cases become complex. 00:07:20.723 --> 00:07:26.482 Adjusting for complexity measured after support begins may remove part of the programme pathway. 00:07:26.602 --> 00:07:33.568 Prefer baseline characteristics when the aim is to compare like with like, and explain the causal role of 00:07:33.568 --> 00:07:34.729 each adjustment variable. 00:07:34.849 --> 00:07:41.070 Check overlap: comparing districts with entirely different case profiles may require extrapolation 00:07:41.070 --> 00:07:48.328 rather than direct evidence. Report raw results alongside adjusted estimates, with the model or 00:07:48.328 --> 00:07:49.884 weighting rule documented. 00:07:50.004 --> 00:07:57.494 Do not present an adjusted ranking as a league table of managerial quality unless the design supports that 00:07:57.494 --> 00:07:57.910 interpretation. 00:07:58.030 --> 00:08:04.849 Adjustment addresses specified differences; it cannot guarantee that all relevant differences have been 00:08:04.849 --> 00:08:06.947 measured or handled correctly. 00:08:07.617 --> 00:08:13.459 A rate from a probability sample has sampling uncertainty because different selected people could 00:08:13.459 --> 00:08:15.128 produce a different estimate. 00:08:15.248 --> 00:08:21.385 A complete administrative count of a defined population does not acquire sampling uncertainty 00:08:21.385 --> 00:08:28.465 merely because it appears on a dashboard. It still faces measurement error, missing records, duplicate 00:08:28.465 --> 00:08:30.353 episodes and delayed updates. 00:08:30.473 --> 00:08:36.742 Distinguish these sources rather than attaching a generic confidence interval to every number. For a 00:08:36.742 --> 00:08:40.504 sampled indicator, use weights and design information when appropriate. 00:08:40.624 --> 00:08:46.789 For an administrative indicator, show reporting coverage, unresolved validation issues and revision 00:08:46.789 --> 00:08:52.954 status. A target line is a management benchmark, not an uncertainty calculation. 00:08:53.074 --> 00:08:59.906 When the displayed value sits close to that line, the evidence quality and consequences of a wrong response 00:08:59.906 --> 00:09:01.804 should influence the action taken. 00:09:02.474 --> 00:09:08.711 Assume a simple random sample of four hundred independent observations from a much larger 00:09:08.711 --> 00:09:10.938 population, with sixty percent successful. 00:09:11.058 --> 00:09:18.054 The standard error of a proportion is approximately the square root of the proportion times one minus the 00:09:18.054 --> 00:09:19.998 proportion, divided by sample size. 00:09:20.118 --> 00:09:27.219 Here, the square root of zero point six times zero point four divided by four hundred is about zero point 00:09:27.219 --> 00:09:28.639 zero two four five. 00:09:28.759 --> 00:09:36.784 Multiply by one point nine six to obtain a margin of roughly four point eight percentage points. The 00:09:36.784 --> 00:09:42.134 approximate interval runs from fifty-five point two to sixty-four point eight percent. 00:09:42.254 --> 00:09:48.243 This normal approximation is reasonable for these illustrative counts, but actual survey designs may 00:09:48.243 --> 00:09:53.377 require different standard errors. Keep that design assumption next to the calculation. 00:09:54.046 --> 00:10:00.503 A dashboard should not jump directly from a changed number to a disciplinary or spending decision. Define 00:10:00.503 --> 00:10:04.681 a signal rule, then an investigation step and an authorised response. 00:10:04.801 --> 00:10:11.549 For example, a sustained backlog increase might trigger a supervisor review of staffing, case 00:10:11.549 --> 00:10:17.815 complexity and reporting completeness. The review determines whether temporary staff, process repair or 00:10:17.815 --> 00:10:19.743 data correction is appropriate. 00:10:19.863 --> 00:10:27.663 Record the action, owner and review date in the same operating system used for follow-up. Also consider 00:10:27.663 --> 00:10:33.627 false alarms when monitoring many indicators repeatedly; isolated threshold crossings can occur by 00:10:33.627 --> 00:10:34.085 chance. 00:10:34.205 --> 00:10:39.755 Use a documented monitoring method suitable for the process and its dependence over time. 00:10:39.875 --> 00:10:47.215 The practical aim is a reliable feedback loop in which evidence changes action and subsequent data show 00:10:47.215 --> 00:10:48.942 whether that action helped. 00:10:49.612 --> 00:10:55.621 Pause and decide whether the sample result supports an automatic green status meaning performance is 00:10:55.621 --> 00:11:01.629 established above the target. Use the interval we just calculated, not the point estimate alone. 00:11:01.749 --> 00:11:08.345 Then consider a survey design effect of two. Multiply the original variance by two, which means multiplying 00:11:08.345 --> 00:11:11.838 the standard error by the square root of two. 00:11:11.958 --> 00:11:17.769 Recalculate the approximate ninety-five percent interval and explain what changed. The design effect 00:11:17.769 --> 00:11:24.473 is supplied for this exercise; in practice it must correspond to the outcome and estimator. 00:11:24.593 --> 00:11:30.582 Finally, propose a useful dashboard label that distinguishes the observed estimate from the strength 00:11:30.582 --> 00:11:32.720 of evidence about the target. 00:11:32.840 --> 00:11:39.052 The goal is a clear management message, not a display that suggests unwarranted certainty through colour. 00:11:39.722 --> 00:11:46.446 The simple random interval includes fifty-eight percent, so the result does not establish that 00:11:46.446 --> 00:11:48.847 underlying performance exceeds the target. 00:11:48.967 --> 00:11:55.387 With a design effect of two, the standard error rises to about three point four six percentage points. 00:11:55.507 --> 00:12:03.135 Multiplying by one point nine six gives a margin of about six point seven nine points, producing an 00:12:03.135 --> 00:12:08.220 interval from fifty-three point two one to sixty-six point seven nine percent. 00:12:08.340 --> 00:12:16.109 The target remains inside, with greater uncertainty. A useful label is: estimate above target, evidence not 00:12:16.109 --> 00:12:16.595 conclusive. 00:12:16.715 --> 00:12:23.151 Management may still act based on risk and cost, but should not confuse that decision with statistical 00:12:23.151 --> 00:12:23.530 proof. 00:12:23.650 --> 00:12:29.114 If repeated monitoring is intended, define a consistent monitoring approach rather than treating 00:12:29.114 --> 00:12:32.056 every new interval as an independent test. 00:12:32.726 --> 00:12:40.088 Place the information needed for interpretation next to the indicator: numerator, denominator, period, 00:12:40.088 --> 00:12:42.919 target definition and data status. 00:12:43.039 --> 00:12:50.308 Use direct labels and text as well as colour so the display remains understandable to people with 00:12:50.308 --> 00:12:52.873 colour-vision differences and on monochrome exports. 00:12:52.993 --> 00:13:00.445 Keep time scales and axes consistent where comparisons depend on them, and avoid truncating a bar axis in a 00:13:00.445 --> 00:13:02.014 way that exaggerates changes. 00:13:02.134 --> 00:13:08.995 Provide a route from the summary to district detail and the metric definition without exposing personal 00:13:08.995 --> 00:13:14.998 records. For revised data, show that a revision occurred and retain a reproducible snapshot. 00:13:15.118 --> 00:13:20.933 A visually restrained dashboard can carry substantial analytical depth when the hierarchy makes the 00:13:20.933 --> 00:13:26.333 decision, the evidence and the uncertainty easy to find in the right order. 00:13:27.003 --> 00:13:34.013 Choose one indicator in your own organisation and rebuild it as a complete decision process. Write its 00:13:34.013 --> 00:13:38.961 metric contract, trace the source records and reproduce its numerator and denominator. 00:13:39.081 --> 00:13:45.151 Test whether a change in population composition explains the recent trend. Identify the appropriate 00:13:45.151 --> 00:13:51.221 uncertainty or data-quality statement and agree an action rule with the person responsible for 00:13:51.221 --> 00:13:51.655 responding. 00:13:51.775 --> 00:13:58.377 Then run a short review meeting using the dashboard: ask what changed, whether the evidence is reliable, 00:13:58.377 --> 00:14:03.037 what mechanism is plausible and what action will be checked next time. 00:14:03.157 --> 00:14:07.371 Keep the resulting decision log. Your deliverable is more than a screen. 00:14:07.491 --> 00:14:12.716 It is a repeatable connection between well-defined measurements, transparent interpretation and 00:14:12.716 --> 00:14:18.416 accountable action that can withstand questions from both analysts and operational managers.