Evaluation design for credible causal claims IDER in-house training materials Hypothetical cases and figures for teaching the methods. Learning objectives Define an estimand and defend a counterfactual comparison. Calculate and interpret DiD and complier effects with their assumptions. Connect uncertainty and incremental costs to a defensible decision. Basic percentages, averages and familiarity with programme monitoring. Design evidence that can support a causal claim Define the policy decision Separate change from impact All case figures are hypothetical Imagine advising a training programme that reports higher earnings after its first year. Your task is to decide whether expansion is justified. Rising earnings are encouraging, but they do not establish that training caused the improvement. Economic recovery, participant selection and other support may explain some change. In this clinic we will build an evaluation argument from a precise policy question through a credible comparison, then interpret effect sizes and uncertainty. Every numerical case is hypothetical. Keep a calculator and a sheet of paper available. Junior practitioners should follow the construction carefully; experienced practitioners should challenge each identifying assumption. The deliverable is a defensible decision memo, supported by an analysis that another researcher could reproduce. Write the estimand before choosing the estimator Population and intervention Outcome and follow-up horizon Offer effect or participation effect? Start by writing a complete sentence describing the effect you want to estimate. For example, what is the effect of offering training to eligible applicants on employment twelve months later? That sentence fixes the population, intervention, outcome and horizon. It also identifies the offer as the policy lever. Comparing people who attended with people who did not attend answers a different question because attendance may reflect motivation or available time. Create an estimand register before analysing outcomes. Include the unit of assignment, unit of observation and the population to which you expect results to apply. If these definitions change later, preserve the original version and explain why. Clear definitions prevent a technically sophisticated model from answering an irrelevant question. Turn a theory of change into measurable tests Training → skills → search → employment Measure delivery and intermediate outcomes Record competing explanations A useful theory of change identifies links that could fail. Training may improve skills, but employment will not rise if firms are not hiring or transport prevents job search. Measure attendance and training quality to establish delivery, then measure skills and search behaviour to investigate mechanisms. Do not treat a positive intermediate outcome as proof of the final effect. Place each measure on a timeline so that it precedes the outcome it is supposed to explain. Also record competing interventions, such as a transport subsidy or wage support. In the analysis plan, distinguish primary outcomes from mechanism checks. This distinction protects the main inference while giving programme managers useful explanations when implementation succeeds but the intended economic benefit does not appear. Choose a counterfactual that survives scrutiny Random assignment when feasible Credible comparison when assignment is observational Document spillovers and selection The counterfactual is the outcome participants would have experienced without the programme. We cannot observe it directly, so design determines whether a comparison can approximate it. With genuine random assignment, retain the original assignment variable and analyse everyone in their assigned group. With observational assignment, investigate the actual selection rule before choosing matching, discontinuity or a differences approach. Matching measured characteristics does not automatically remove unmeasured motivation. Ask whether comparison units receive related support or learn from treated neighbours; such spillovers change the contrast being estimated. Document access restrictions, consent and the allocation process. A design diagram showing who enters each group and when treatment begins is often more useful than an early list of regression specifications. Difference in differences removes a shared change Treated profit: 100 → 140 Comparison profit: 80 → 100 Estimated additional change: 20 (140 − 100) − (100 − 80) = 20 Consider average monthly profit measured in the same monetary units for two groups. Treated businesses increase from one hundred to one hundred and forty. Comparison businesses increase from eighty to one hundred. The treated change is forty, but twenty units of improvement also occurred without the programme. Subtracting the comparison change gives an estimated effect of twenty. Another way to see this is to add the comparison change to the treated baseline: one hundred plus twenty predicts a counterfactual profit of one hundred and twenty. The observed treated outcome exceeds that prediction by twenty. The arithmetic is easy; credibility depends on whether untreated trends would have been parallel, together with stable measurement, appropriate composition and no differential concurrent shock. Interrogate parallel trends and treatment timing Inspect multiple pre-treatment periods Investigate differential shocks Use timing-appropriate comparisons Parallel trends concerns the changes that would occur without treatment, not equal starting levels. Examine several pre-treatment periods and explain why the groups face comparable economic forces. Similar historical movements support the argument but cannot prove the future counterfactual. Check whether firms enter or leave the sample differently and whether measurement changes coincide with treatment. When programme adoption occurs at different times, avoid automatically using already treated units as clean controls for newly treated units. Effects may vary by cohort and time since treatment. Estimate and aggregate appropriate group and time effects using explicit comparison rules. Before trusting a single coefficient, draw the timing matrix and identify which observations supply each counterfactual. This exposes hidden comparisons that a regression table can obscure. Calculate the effect and challenge attribution Treated employment: 50% → 68% Comparison employment: 40% → 50% Only treated areas receive a transport subsidy Pause here and write three answers. First, calculate the difference in differences in employment, expressing it in percentage points. Second, calculate the treated group's predicted employment without the programme under the parallel trends assumption. Third, explain how a transport subsidy introduced only in treated areas affects the claim that training caused the result. Work from changes rather than subtracting only the final percentages. After calculating the absolute effect, you may also express it relative to the counterfactual employment rate, but label that denominator explicitly. Do not try to repair the subsidy problem by simply adding more decimal places. The exercise asks you to distinguish a correct numerical contrast from a credible interpretation of the intervention being evaluated. Eight points is a contrast, not automatic attribution 18 − 10 = 8 percentage points Counterfactual employment: 60% Concurrent subsidy threatens training-only attribution The treated increase is eighteen percentage points and the comparison increase is ten, leaving eight percentage points. Under the assumed shared trend, treated employment would have reached sixty percent. Eight divided by sixty gives about thirteen point three three percent relative improvement; this is different from an eight percent increase. Now consider the subsidy. If only treated areas receive it at the same time, the estimated contrast can combine training and transport effects. You cannot isolate the training effect with this simple comparison. Seek a design that separately varies the interventions or narrow the claim to the package actually evaluated, while assessing remaining assumptions. In the memo, show the arithmetic and the attribution limitation beside each other so neither is lost. Preserve assignment when take-up is imperfect Offer effect: intention to treat Attendance comparisons can be selected Track take-up in both assigned groups Suppose some people offered training decline and some comparison participants obtain equivalent training elsewhere. The policy effect of offering the programme remains meaningful. Estimate it by comparing outcomes according to original assignment, regardless of attendance. This is the intention to treat effect. Excluding non-attenders can destroy random comparability because attendance is a choice influenced by constraints and expected benefits. Report take-up in both groups, including the definition and measurement source. Then decide whether the practical question requires an additional participation effect. A programme manager choosing whether to offer places often needs the offer effect directly. Do not portray imperfect take-up as a statistical nuisance to erase; it can reveal delivery barriers that determine how the intervention will work at scale. A participation effect requires additional assumptions Offer effect on outcome: 12 units Take-up: 70% offered; 10% comparison 12 ÷ (0.70 − 0.10) = 20 12 ÷ (0.70 − 0.10) = 20 To illustrate an instrumental variables calculation, suppose random assignment increases the outcome by twelve units and raises participation from ten to seventy percent. The first stage is sixty percentage points, or zero point six. Dividing twelve by zero point six yields twenty outcome units for people whose participation is induced by the offer. This local interpretation requires a relevant, independent assignment, no people whose participation is reduced by the offer, and an exclusion restriction: assignment affects the outcome only through participation. An offer that also provides cash or encouragement with its own effect can violate exclusion. Therefore, report the assumptions and first stage with the result. Twenty is not automatically the effect for all participants, all eligible people or a future national rollout. Match uncertainty to the assignment structure Cluster where dependence requires it Report effect and interval together Distinguish statistical and practical importance Uncertainty is about the amount of information in the design, not simply the number of rows in a spreadsheet. If villages are assigned together and their outcomes share shocks, treating every person as an independent assignment can understate uncertainty. Use an inference method appropriate to the assignment and sampling structure, paying particular attention to small numbers of clusters. Report the effect in interpretable units with a confidence interval. Then compare that interval with a prespecified minimum worthwhile effect. An estimate may be distinguishable from zero but still too imprecise to establish a meaningful benefit. Conversely, an interval crossing zero does not prove that the programme has no effect. State which decision-relevant values remain compatible with the evidence and what additional information would help. Plan checks that can change your interpretation Audit missing outcomes and attrition Prespecify outcomes and subgroup tests Investigate mechanisms without overclaiming A robustness section should investigate concrete threats rather than display many similar regressions. Start with a participant flow table showing assigned, contacted and observed cases by group. If missing outcomes differ, explain possible selection and examine justified sensitivity analyses. A balanced observed sample cannot by itself prove that attrition is harmless. Next, distinguish prespecified outcomes from exploratory findings and account for multiple testing when making families of claims. For subgroup analysis, test whether estimated effects differ; one subgroup being statistically significant and another not significant does not establish a difference. Finally, compare results with implementation evidence. If delivery was weak, the estimate may describe the actual offer under weak implementation rather than the effect of fully delivered training. Preserve that policy distinction. Does a positive estimate clear the decision threshold? Estimated employment effect: 8 percentage points Standard error: 3 points Minimum worthwhile effect: 5 points Pause and calculate an approximate ninety-five percent confidence interval using the estimate plus or minus one point nine six times its standard error. Keep every quantity in percentage points. Next, answer two separate questions: is zero excluded, and is the minimum worthwhile effect of five points excluded on the low side? Write a recommendation that recognises the difference. Assume the supplied standard error already reflects an appropriate design and a normal approximation is justified. In real work, that assumption requires checking, especially with few clusters. You are practising interpretation rather than selecting the inference method here. Before continuing, identify one additional piece of information a decision maker would need, such as incremental cost, implementation capacity or the consequences of delaying a decision. Evidence of a benefit does not establish sufficient benefit 95% interval: 2.12 to 13.88 points Zero is excluded Benefits below 5 points remain plausible Multiplying three by one point nine six gives five point eight eight percentage points. Subtracting and adding that margin to eight produces an interval from two point one two to thirteen point eight eight. Zero is outside, so this calculation provides evidence of a positive effect under these assumptions. However, the interval includes effects below five points. It therefore does not establish that the programme exceeds the minimum worthwhile effect. A defensible recommendation could support a bounded expansion with further measurement if costs and risks justify it, but that is a decision under uncertainty, not a statistical conclusion. Do not interpret the interval as a ninety-five percent probability that this fixed effect lies inside these observed endpoints; its coverage describes repeated use of the procedure. Translate a valid effect into an incremental cost metric Offer effect: 0.08 × 1,000 = 80 Incremental programme cost: 240,000 Cost per additional employed person: 3,000 240,000 ÷ (0.08 × 1,000) = 3,000 Suppose the offer effect is eight percentage points across one thousand people offered the programme. Multiplying gives an estimated eighty additional people employed at the measured follow-up date. If incremental programme cost is two hundred and forty thousand monetary units, the ratio is three thousand per additional employed person. Use incremental costs relative to the comparison, with a clearly stated perspective and price year. Do not divide cost by all observed employed participants, because many would have been employed anyway. Employment at one date is also different from sustained employment or lifetime earnings. Explore how the ratio changes across plausible effects and costs, and compare alternatives measured on a consistent basis. A cost-effectiveness ratio informs a decision; it does not replace distributional or implementation judgement. Build a claim that another practitioner can audit Estimand, design and assumptions Reproducible estimate and sensitivity checks Decision, limits and next evidence step Prepare a short evaluation memo organised around the claim you can defend. State the estimand and describe how assignment or comparison creates the counterfactual. Show participant flow, measurement definitions and the main effect with its uncertainty. Attach reproducible code, a data dictionary and a record of deviations from the analysis plan. Explain the most consequential threats in plain language, including concurrent interventions, spillovers, attrition and limits to transferability. Finish with the decision that follows at current costs and the evidence that would change it. For your own practice, select a recent programme report and rewrite one causal claim at the level its design supports. The objective is disciplined judgement: technically correct calculations joined to an honest explanation of what the evidence identifies. Check your understanding 1. Employment changes from 50% to 68% in treated areas and 40% to 50% in comparison areas. What is the difference in differences? A. 8 percentage points B. 18 percentage points C. 28 percentage points D. 8% relative to the counterfactual 2. A transport subsidy begins only in treated areas at the same time as training. What is the main implication? A. It automatically improves statistical precision B. The contrast cannot isolate training from the concurrent subsidy without further design assumptions C. It proves parallel trends D. It can be ignored if the sample is large 3. Why analyse original random assignment when some invitees do not attend? A. To estimate the effect of offering training while preserving the assigned comparison B. To remove non-attenders from the sample C. To guarantee an effect for every participant D. To eliminate all spillovers 4. The offer effect is 12 units and take-up is 70% versus 10%. Under valid IV assumptions, what is estimated? A. 12 units for all eligible people B. 20 units for people induced to participate by the offer C. 17.14 units for all participants D. 60 units for non-participants 5. An effect of 8 points has a 95% interval of 2.12–13.88 points. The minimum worthwhile effect is 5 points. Which statement is supported? A. The benefit certainly exceeds 5 points B. No benefit exists C. Zero is excluded, but effects below the worthwhile threshold remain compatible with the interval D. The effect equals exactly 8 points 6. An offer effect of 0.08 applies to 1,000 offers. Incremental cost is 240,000. What is cost per additional employed person? A. 240 B. 1,920 C. 3,000 D. 30,000 Applied task 1 Design a one-page evaluation of a training offer before rollout. Specify the estimand, comparison, measurement dates and the two greatest threats. Applied task 2 Write a decision note using an 8-point effect, standard error 3 points, 1,000 offers and incremental cost 240,000. The worthwhile threshold is 5 points. Answers and explanations 1. A. The treated change is 18 points; subtract the comparison change of 10 points to obtain 8 percentage points. (Slide 8) 2. B. A differential concurrent intervention threatens attribution to training alone; a larger sample does not remove this identification problem. (Slide 8) 3. A. The intention to treat contrast compares assigned groups and estimates the policy effect of the offer, including actual take-up behaviour. (Slide 9) 4. B. The first stage is 0.60; 12/0.60 = 20. This is a local effect for compliers, not automatically everyone. (Slide 10) 5. C. A positive interval does not establish that the effect exceeds every positive decision threshold. (Slide 14) 6. C. Estimated additional employment is 0.08 × 1,000 = 80; 240,000/80 = 3,000. (Slide 15) Applied task 1 A strong design defines eligible applicants, the effect of an offer on employment at twelve months, and a feasible allocation or justified comparison rule. It measures baseline and follow-up consistently, records concurrent support, preserves assignment, and plans attrition and spillover checks. Alternative designs are acceptable when their identifying assumptions are explicit. Estimand fixes population, lever, outcome and horizon. Counterfactual and assignment or selection mechanism are explained. Threats have specific measurement or design responses. Applied task 2 The approximate 95% interval is 2.12–13.88 points; the point estimate implies 80 additional employed people and 3,000 per additional person. Evidence supports a positive effect but does not establish exceeding 5 points. Recommend an action conditional on costs, risk and implementation capacity, and identify what further evidence would change it. Correct interval and additional-outcome calculation. Distinguishes positive evidence from exceeding the threshold. Recommendation acknowledges uncertainty and incremental cost. References J-PAL — Data analysis https://www.povertyactionlab.org/resource/data-analysis Definitions of offer and complier effects; interpretation of instrumental-variable assumptions. Callaway and Sant’Anna — Introduction to DiD with Multiple Time Periods https://bcallaway11.github.io/did/articles/multi-period-did.html Timing-appropriate comparison groups and heterogeneous effects in staggered adoption. J-PAL — Pre-analysis plans https://www.povertyactionlab.org/resource/pre-analysis-plans Prespecification of outcomes and transparent deviations. J-PAL — Introduction to randomized evaluations https://www.povertyactionlab.org/resource/introduction-randomized-evaluations Counterfactual reasoning and assignment-based evaluation design.