WEBVTT NOTE IDER in-house training materials. Captions follow measured phrase audio timings. Hypothetical worked cases; figures are instructional examples. 00:00:00.000 --> 00:00:06.881 Imagine advising a training programme that reports higher earnings after its first year. Your task is to 00:00:06.881 --> 00:00:08.905 decide whether expansion is justified. 00:00:09.025 --> 00:00:14.889 Rising earnings are encouraging, but they do not establish that training caused the improvement. 00:00:14.889 --> 00:00:19.497 Economic recovery, participant selection and other support may explain some change. 00:00:19.617 --> 00:00:27.280 In this clinic we will build an evaluation argument from a precise policy question through a credible 00:00:27.280 --> 00:00:32.690 comparison, then interpret effect sizes and uncertainty. Every numerical case is hypothetical. 00:00:32.810 --> 00:00:39.791 Keep a calculator and a sheet of paper available. Junior practitioners should follow the construction 00:00:39.791 --> 00:00:43.514 carefully; experienced practitioners should challenge each identifying assumption. 00:00:43.634 --> 00:00:50.034 The deliverable is a defensible decision memo, supported by an analysis that another researcher could 00:00:50.034 --> 00:00:50.461 reproduce. 00:00:51.131 --> 00:00:57.825 Start by writing a complete sentence describing the effect you want to estimate. For example, what is the 00:00:57.825 --> 00:01:02.288 effect of offering training to eligible applicants on employment twelve months later? 00:01:02.408 --> 00:01:09.790 That sentence fixes the population, intervention, outcome and horizon. It also identifies the offer as 00:01:09.790 --> 00:01:11.267 the policy lever. 00:01:11.387 --> 00:01:17.529 Comparing people who attended with people who did not attend answers a different question because attendance 00:01:17.529 --> 00:01:22.520 may reflect motivation or available time. Create an estimand register before analysing outcomes. 00:01:22.640 --> 00:01:29.154 Include the unit of assignment, unit of observation and the population to which you expect results to 00:01:29.154 --> 00:01:34.134 apply. If these definitions change later, preserve the original version and explain why. 00:01:34.254 --> 00:01:40.013 Clear definitions prevent a technically sophisticated model from answering an irrelevant question. 00:01:40.683 --> 00:01:46.808 A useful theory of change identifies links that could fail. Training may improve skills, but employment will 00:01:46.808 --> 00:01:51.132 not rise if firms are not hiring or transport prevents job search. 00:01:51.252 --> 00:01:57.585 Measure attendance and training quality to establish delivery, then measure skills and search behaviour to 00:01:57.585 --> 00:02:03.918 investigate mechanisms. Do not treat a positive intermediate outcome as proof of the final effect. 00:02:04.038 --> 00:02:11.074 Place each measure on a timeline so that it precedes the outcome it is supposed to explain. Also record 00:02:11.074 --> 00:02:14.778 competing interventions, such as a transport subsidy or wage support. 00:02:14.898 --> 00:02:19.994 In the analysis plan, distinguish primary outcomes from mechanism checks. 00:02:20.114 --> 00:02:26.591 This distinction protects the main inference while giving programme managers useful explanations when 00:02:26.591 --> 00:02:31.573 implementation succeeds but the intended economic benefit does not appear. 00:02:32.243 --> 00:02:38.239 The counterfactual is the outcome participants would have experienced without the programme. We cannot 00:02:38.239 --> 00:02:43.377 observe it directly, so design determines whether a comparison can approximate it. 00:02:43.497 --> 00:02:50.146 With genuine random assignment, retain the original assignment variable and analyse everyone in their 00:02:50.146 --> 00:02:55.845 assigned group. With observational assignment, investigate the actual selection rule before choosing 00:02:55.845 --> 00:02:58.695 matching, discontinuity or a differences approach. 00:02:58.815 --> 00:03:03.778 Matching measured characteristics does not automatically remove unmeasured motivation. Ask 00:03:03.778 --> 00:03:10.727 whether comparison units receive related support or learn from treated neighbours; such spillovers change 00:03:10.727 --> 00:03:12.712 the contrast being estimated. 00:03:12.832 --> 00:03:18.436 Document access restrictions, consent and the allocation process. A design diagram showing who 00:03:18.436 --> 00:03:25.764 enters each group and when treatment begins is often more useful than an early list of regression 00:03:25.764 --> 00:03:26.195 specifications. 00:03:26.865 --> 00:03:32.704 Consider average monthly profit measured in the same monetary units for two groups. Treated businesses 00:03:32.704 --> 00:03:38.933 increase from one hundred to one hundred and forty. Comparison businesses increase from eighty to one 00:03:38.933 --> 00:03:39.323 hundred. 00:03:39.443 --> 00:03:45.691 The treated change is forty, but twenty units of improvement also occurred without the programme. 00:03:45.691 --> 00:03:49.857 Subtracting the comparison change gives an estimated effect of twenty. 00:03:49.977 --> 00:03:56.000 Another way to see this is to add the comparison change to the treated baseline: one hundred plus 00:03:56.000 --> 00:03:59.346 twenty predicts a counterfactual profit of one hundred and twenty. 00:03:59.466 --> 00:04:03.019 The observed treated outcome exceeds that prediction by twenty. 00:04:03.139 --> 00:04:09.631 The arithmetic is easy; credibility depends on whether untreated trends would have been parallel, together 00:04:09.631 --> 00:04:13.959 with stable measurement, appropriate composition and no differential concurrent shock. 00:04:14.629 --> 00:04:20.950 Parallel trends concerns the changes that would occur without treatment, not equal starting levels. Examine 00:04:20.950 --> 00:04:26.007 several pre-treatment periods and explain why the groups face comparable economic forces. 00:04:26.127 --> 00:04:32.181 Similar historical movements support the argument but cannot prove the future counterfactual. Check whether 00:04:32.181 --> 00:04:38.236 firms enter or leave the sample differently and whether measurement changes coincide with treatment. 00:04:38.356 --> 00:04:43.793 When programme adoption occurs at different times, avoid automatically using already treated units as 00:04:43.793 --> 00:04:49.618 clean controls for newly treated units. Effects may vary by cohort and time since treatment. 00:04:49.738 --> 00:04:55.872 Estimate and aggregate appropriate group and time effects using explicit comparison rules. Before 00:04:55.872 --> 00:05:02.478 trusting a single coefficient, draw the timing matrix and identify which observations supply each 00:05:02.478 --> 00:05:02.950 counterfactual. 00:05:03.070 --> 00:05:06.948 This exposes hidden comparisons that a regression table can obscure. 00:05:07.618 --> 00:05:14.615 Pause here and write three answers. First, calculate the difference in differences in employment, 00:05:14.615 --> 00:05:17.115 expressing it in percentage points. 00:05:17.235 --> 00:05:23.725 Second, calculate the treated group's predicted employment without the programme under the parallel 00:05:23.725 --> 00:05:24.723 trends assumption. 00:05:24.843 --> 00:05:31.024 Third, explain how a transport subsidy introduced only in treated areas affects the claim that training 00:05:31.024 --> 00:05:36.047 caused the result. Work from changes rather than subtracting only the final percentages. 00:05:36.167 --> 00:05:42.917 After calculating the absolute effect, you may also express it relative to the counterfactual employment 00:05:42.917 --> 00:05:45.617 rate, but label that denominator explicitly. 00:05:45.737 --> 00:05:49.998 Do not try to repair the subsidy problem by simply adding more decimal places. 00:05:50.118 --> 00:05:56.635 The exercise asks you to distinguish a correct numerical contrast from a credible interpretation of 00:05:56.635 --> 00:05:58.373 the intervention being evaluated. 00:05:59.043 --> 00:06:05.462 The treated increase is eighteen percentage points and the comparison increase is ten, leaving eight 00:06:05.462 --> 00:06:11.454 percentage points. Under the assumed shared trend, treated employment would have reached sixty percent. 00:06:11.574 --> 00:06:18.254 Eight divided by sixty gives about thirteen point three three percent relative improvement; this is 00:06:18.254 --> 00:06:22.708 different from an eight percent increase. Now consider the subsidy. 00:06:22.828 --> 00:06:29.616 If only treated areas receive it at the same time, the estimated contrast can combine training and transport 00:06:29.616 --> 00:06:33.764 effects. You cannot isolate the training effect with this simple comparison. 00:06:33.884 --> 00:06:41.140 Seek a design that separately varies the interventions or narrow the claim to the package actually evaluated, 00:06:41.140 --> 00:06:42.847 while assessing remaining assumptions. 00:06:42.967 --> 00:06:49.039 In the memo, show the arithmetic and the attribution limitation beside each other so neither is lost. 00:06:49.709 --> 00:06:55.464 Suppose some people offered training decline and some comparison participants obtain equivalent training 00:06:55.464 --> 00:06:59.891 elsewhere. The policy effect of offering the programme remains meaningful. 00:07:00.011 --> 00:07:05.478 Estimate it by comparing outcomes according to original assignment, regardless of attendance. This is 00:07:05.478 --> 00:07:07.430 the intention to treat effect. 00:07:07.550 --> 00:07:13.073 Excluding non-attenders can destroy random comparability because attendance is a choice 00:07:13.073 --> 00:07:19.601 influenced by constraints and expected benefits. Report take-up in both groups, including the 00:07:19.601 --> 00:07:21.610 definition and measurement source. 00:07:21.730 --> 00:07:27.984 Then decide whether the practical question requires an additional participation effect. A programme manager 00:07:27.984 --> 00:07:32.898 choosing whether to offer places often needs the offer effect directly. 00:07:33.018 --> 00:07:38.926 Do not portray imperfect take-up as a statistical nuisance to erase; it can reveal delivery barriers 00:07:38.926 --> 00:07:42.248 that determine how the intervention will work at scale. 00:07:42.918 --> 00:07:49.743 To illustrate an instrumental variables calculation, suppose random assignment increases the outcome by 00:07:49.743 --> 00:07:54.993 twelve units and raises participation from ten to seventy percent. 00:07:55.113 --> 00:08:02.364 The first stage is sixty percentage points, or zero point six. Dividing twelve by zero point six yields 00:08:02.364 --> 00:08:07.199 twenty outcome units for people whose participation is induced by the offer. 00:08:07.319 --> 00:08:12.941 This local interpretation requires a relevant, independent assignment, no people whose participation 00:08:12.941 --> 00:08:19.969 is reduced by the offer, and an exclusion restriction: assignment affects the outcome only through 00:08:19.969 --> 00:08:20.438 participation. 00:08:20.558 --> 00:08:27.832 An offer that also provides cash or encouragement with its own effect can violate exclusion. Therefore, 00:08:27.832 --> 00:08:31.924 report the assumptions and first stage with the result. 00:08:32.044 --> 00:08:38.458 Twenty is not automatically the effect for all participants, all eligible people or a future national 00:08:38.458 --> 00:08:38.859 rollout. 00:08:39.529 --> 00:08:45.194 Uncertainty is about the amount of information in the design, not simply the number of rows in a 00:08:45.194 --> 00:08:45.508 spreadsheet. 00:08:45.628 --> 00:08:52.474 If villages are assigned together and their outcomes share shocks, treating every person as an independent 00:08:52.474 --> 00:08:54.185 assignment can understate uncertainty. 00:08:54.305 --> 00:09:00.496 Use an inference method appropriate to the assignment and sampling structure, paying particular attention to 00:09:00.496 --> 00:09:06.275 small numbers of clusters. Report the effect in interpretable units with a confidence interval. 00:09:06.395 --> 00:09:11.773 Then compare that interval with a prespecified minimum worthwhile effect. An estimate may be distinguishable 00:09:11.773 --> 00:09:15.718 from zero but still too imprecise to establish a meaningful benefit. 00:09:15.838 --> 00:09:22.224 Conversely, an interval crossing zero does not prove that the programme has no effect. State which 00:09:22.224 --> 00:09:27.413 decision-relevant values remain compatible with the evidence and what additional information would help. 00:09:28.083 --> 00:09:33.539 A robustness section should investigate concrete threats rather than display many similar regressions. 00:09:33.539 --> 00:09:39.414 Start with a participant flow table showing assigned, contacted and observed cases by group. 00:09:39.534 --> 00:09:46.374 If missing outcomes differ, explain possible selection and examine justified sensitivity analyses. A balanced 00:09:46.374 --> 00:09:51.260 observed sample cannot by itself prove that attrition is harmless. 00:09:51.380 --> 00:09:57.010 Next, distinguish prespecified outcomes from exploratory findings and account for multiple testing 00:09:57.010 --> 00:09:59.356 when making families of claims. 00:09:59.476 --> 00:10:06.101 For subgroup analysis, test whether estimated effects differ; one subgroup being statistically significant 00:10:06.101 --> 00:10:12.726 and another not significant does not establish a difference. Finally, compare results with 00:10:12.726 --> 00:10:13.745 implementation evidence. 00:10:13.865 --> 00:10:19.998 If delivery was weak, the estimate may describe the actual offer under weak implementation rather than the 00:10:19.998 --> 00:10:23.246 effect of fully delivered training. Preserve that policy distinction. 00:10:23.916 --> 00:10:30.719 Pause and calculate an approximate ninety-five percent confidence interval using the estimate plus or minus 00:10:30.719 --> 00:10:37.070 one point nine six times its standard error. Keep every quantity in percentage points. 00:10:37.190 --> 00:10:43.635 Next, answer two separate questions: is zero excluded, and is the minimum worthwhile effect of five points 00:10:43.635 --> 00:10:48.184 excluded on the low side? Write a recommendation that recognises the difference. 00:10:48.304 --> 00:10:55.174 Assume the supplied standard error already reflects an appropriate design and a normal approximation is 00:10:55.174 --> 00:11:00.669 justified. In real work, that assumption requires checking, especially with few clusters. 00:11:00.789 --> 00:11:04.655 You are practising interpretation rather than selecting the inference method here. 00:11:04.775 --> 00:11:11.815 Before continuing, identify one additional piece of information a decision maker would need, such as 00:11:11.815 --> 00:11:16.977 incremental cost, implementation capacity or the consequences of delaying a decision. 00:11:17.647 --> 00:11:23.220 Multiplying three by one point nine six gives five point eight eight percentage points. 00:11:23.340 --> 00:11:29.066 Subtracting and adding that margin to eight produces an interval from two point one two to thirteen point 00:11:29.066 --> 00:11:29.702 eight eight. 00:11:29.822 --> 00:11:36.574 Zero is outside, so this calculation provides evidence of a positive effect under these assumptions. However, 00:11:36.574 --> 00:11:39.528 the interval includes effects below five points. 00:11:39.648 --> 00:11:44.106 It therefore does not establish that the programme exceeds the minimum worthwhile effect. 00:11:44.226 --> 00:11:50.894 A defensible recommendation could support a bounded expansion with further measurement if costs and risks 00:11:50.894 --> 00:11:56.672 justify it, but that is a decision under uncertainty, not a statistical conclusion. 00:11:56.792 --> 00:12:03.861 Do not interpret the interval as a ninety-five percent probability that this fixed effect lies inside these 00:12:03.861 --> 00:12:08.019 observed endpoints; its coverage describes repeated use of the procedure. 00:12:08.689 --> 00:12:14.337 Suppose the offer effect is eight percentage points across one thousand people offered the programme. 00:12:14.337 --> 00:12:19.231 Multiplying gives an estimated eighty additional people employed at the measured follow-up date. 00:12:19.351 --> 00:12:26.080 If incremental programme cost is two hundred and forty thousand monetary units, the ratio is three thousand 00:12:26.080 --> 00:12:27.664 per additional employed person. 00:12:27.784 --> 00:12:33.813 Use incremental costs relative to the comparison, with a clearly stated perspective and price year. Do not 00:12:33.813 --> 00:12:38.778 divide cost by all observed employed participants, because many would have been employed anyway. 00:12:38.898 --> 00:12:45.202 Employment at one date is also different from sustained employment or lifetime earnings. Explore how 00:12:45.202 --> 00:12:51.505 the ratio changes across plausible effects and costs, and compare alternatives measured on a consistent 00:12:51.505 --> 00:12:51.925 basis. 00:12:52.045 --> 00:12:58.826 A cost-effectiveness ratio informs a decision; it does not replace distributional or implementation 00:12:58.826 --> 00:12:59.347 judgement. 00:13:00.017 --> 00:13:07.592 Prepare a short evaluation memo organised around the claim you can defend. State the estimand and describe 00:13:07.592 --> 00:13:10.710 how assignment or comparison creates the counterfactual. 00:13:10.830 --> 00:13:17.115 Show participant flow, measurement definitions and the main effect with its uncertainty. Attach reproducible 00:13:17.115 --> 00:13:22.951 code, a data dictionary and a record of deviations from the analysis plan. 00:13:23.071 --> 00:13:29.667 Explain the most consequential threats in plain language, including concurrent interventions, 00:13:29.667 --> 00:13:33.265 spillovers, attrition and limits to transferability. 00:13:33.385 --> 00:13:37.901 Finish with the decision that follows at current costs and the evidence that would change it. 00:13:38.021 --> 00:13:44.520 For your own practice, select a recent programme report and rewrite one causal claim at the level its 00:13:44.520 --> 00:13:45.242 design supports. 00:13:45.362 --> 00:13:50.972 The objective is disciplined judgement: technically correct calculations joined to an honest explanation 00:13:50.972 --> 00:13:53.129 of what the evidence identifies.