Evaluation design for credible causal claims
Produced: 2026-09-20
Language: en
IDER in-house training materials: practical cases, illustrations and guided exercises.
All cases and figures are hypothetical teaching examples, not current market data.

[00:00:00.000 – 00:00:51.011]
Design evidence that can support a causal claim
ON SCREEN
Define the policy decision
Separate change from impact
All case figures are hypothetical

TEACHING COMMENTARY
Imagine advising a training programme that reports higher earnings after its first year. Your task is to decide whether expansion is justified. Rising earnings are encouraging, but they do not establish that training caused the improvement. Economic recovery, participant selection and other support may explain some change. In this clinic we will build an evaluation argument from a precise policy question through a credible comparison, then interpret effect sizes and uncertainty. Every numerical case is hypothetical. Keep a calculator and a sheet of paper available. Junior practitioners should follow the construction carefully; experienced practitioners should challenge each identifying assumption. The deliverable is a defensible decision memo, supported by an analysis that another researcher could reproduce.

[00:00:51.011 – 00:01:40.563]
Write the estimand before choosing the estimator
ON SCREEN
Population and intervention
Outcome and follow-up horizon
Offer effect or participation effect?

TEACHING COMMENTARY
Start by writing a complete sentence describing the effect you want to estimate. For example, what is the effect of offering training to eligible applicants on employment twelve months later? That sentence fixes the population, intervention, outcome and horizon. It also identifies the offer as the policy lever. Comparing people who attended with people who did not attend answers a different question because attendance may reflect motivation or available time. Create an estimand register before analysing outcomes. Include the unit of assignment, unit of observation and the population to which you expect results to apply. If these definitions change later, preserve the original version and explain why. Clear definitions prevent a technically sophisticated model from answering an irrelevant question.

[00:01:40.563 – 00:02:32.123]
Turn a theory of change into measurable tests
ON SCREEN
Training → skills → search → employment
Measure delivery and intermediate outcomes
Record competing explanations

TEACHING COMMENTARY
A useful theory of change identifies links that could fail. Training may improve skills, but employment will not rise if firms are not hiring or transport prevents job search. Measure attendance and training quality to establish delivery, then measure skills and search behaviour to investigate mechanisms. Do not treat a positive intermediate outcome as proof of the final effect. Place each measure on a timeline so that it precedes the outcome it is supposed to explain. Also record competing interventions, such as a transport subsidy or wage support. In the analysis plan, distinguish primary outcomes from mechanism checks. This distinction protects the main inference while giving programme managers useful explanations when implementation succeeds but the intended economic benefit does not appear.

[00:02:32.123 – 00:03:26.745]
Choose a counterfactual that survives scrutiny
ON SCREEN
Random assignment when feasible
Credible comparison when assignment is observational
Document spillovers and selection

TEACHING COMMENTARY
The counterfactual is the outcome participants would have experienced without the programme. We cannot observe it directly, so design determines whether a comparison can approximate it. With genuine random assignment, retain the original assignment variable and analyse everyone in their assigned group. With observational assignment, investigate the actual selection rule before choosing matching, discontinuity or a differences approach. Matching measured characteristics does not automatically remove unmeasured motivation. Ask whether comparison units receive related support or learn from treated neighbours; such spillovers change the contrast being estimated. Document access restrictions, consent and the allocation process. A design diagram showing who enters each group and when treatment begins is often more useful than an early list of regression specifications.

[00:03:26.745 – 00:04:14.509]
Difference in differences removes a shared change
ON SCREEN
Treated profit: 100 → 140
Comparison profit: 80 → 100
Estimated additional change: 20
(140 − 100) − (100 − 80) = 20
TEACHING COMMENTARY
Consider average monthly profit measured in the same monetary units for two groups. Treated businesses increase from one hundred to one hundred and forty. Comparison businesses increase from eighty to one hundred. The treated change is forty, but twenty units of improvement also occurred without the programme. Subtracting the comparison change gives an estimated effect of twenty. Another way to see this is to add the comparison change to the treated baseline: one hundred plus twenty predicts a counterfactual profit of one hundred and twenty. The observed treated outcome exceeds that prediction by twenty. The arithmetic is easy; credibility depends on whether untreated trends would have been parallel, together with stable measurement, appropriate composition and no differential concurrent shock.

[00:04:14.509 – 00:05:07.498]
Interrogate parallel trends and treatment timing
ON SCREEN
Inspect multiple pre-treatment periods
Investigate differential shocks
Use timing-appropriate comparisons

TEACHING COMMENTARY
Parallel trends concerns the changes that would occur without treatment, not equal starting levels. Examine several pre-treatment periods and explain why the groups face comparable economic forces. Similar historical movements support the argument but cannot prove the future counterfactual. Check whether firms enter or leave the sample differently and whether measurement changes coincide with treatment. When programme adoption occurs at different times, avoid automatically using already treated units as clean controls for newly treated units. Effects may vary by cohort and time since treatment. Estimate and aggregate appropriate group and time effects using explicit comparison rules. Before trusting a single coefficient, draw the timing matrix and identify which observations supply each counterfactual. This exposes hidden comparisons that a regression table can obscure.

[00:05:07.498 – 00:05:58.923]
Calculate the effect and challenge attribution
ON SCREEN
Treated employment: 50% → 68%
Comparison employment: 40% → 50%
Only treated areas receive a transport subsidy

TEACHING COMMENTARY
Pause here and write three answers. First, calculate the difference in differences in employment, expressing it in percentage points. Second, calculate the treated group's predicted employment without the programme under the parallel trends assumption. Third, explain how a transport subsidy introduced only in treated areas affects the claim that training caused the result. Work from changes rather than subtracting only the final percentages. After calculating the absolute effect, you may also express it relative to the counterfactual employment rate, but label that denominator explicitly. Do not try to repair the subsidy problem by simply adding more decimal places. The exercise asks you to distinguish a correct numerical contrast from a credible interpretation of the intervention being evaluated.

[00:05:58.923 – 00:06:49.589]
Eight points is a contrast, not automatic attribution
ON SCREEN
18 − 10 = 8 percentage points
Counterfactual employment: 60%
Concurrent subsidy threatens training-only attribution

TEACHING COMMENTARY
The treated increase is eighteen percentage points and the comparison increase is ten, leaving eight percentage points. Under the assumed shared trend, treated employment would have reached sixty percent. Eight divided by sixty gives about thirteen point three three percent relative improvement; this is different from an eight percent increase. Now consider the subsidy. If only treated areas receive it at the same time, the estimated contrast can combine training and transport effects. You cannot isolate the training effect with this simple comparison. Seek a design that separately varies the interventions or narrow the claim to the package actually evaluated, while assessing remaining assumptions. In the memo, show the arithmetic and the attribution limitation beside each other so neither is lost.

[00:06:49.589 – 00:07:42.798]
Preserve assignment when take-up is imperfect
ON SCREEN
Offer effect: intention to treat
Attendance comparisons can be selected
Track take-up in both assigned groups

TEACHING COMMENTARY
Suppose some people offered training decline and some comparison participants obtain equivalent training elsewhere. The policy effect of offering the programme remains meaningful. Estimate it by comparing outcomes according to original assignment, regardless of attendance. This is the intention to treat effect. Excluding non-attenders can destroy random comparability because attendance is a choice influenced by constraints and expected benefits. Report take-up in both groups, including the definition and measurement source. Then decide whether the practical question requires an additional participation effect. A programme manager choosing whether to offer places often needs the offer effect directly. Do not portray imperfect take-up as a statistical nuisance to erase; it can reveal delivery barriers that determine how the intervention will work at scale.

[00:07:42.798 – 00:08:39.409]
A participation effect requires additional assumptions
ON SCREEN
Offer effect on outcome: 12 units
Take-up: 70% offered; 10% comparison
12 ÷ (0.70 − 0.10) = 20
12 ÷ (0.70 − 0.10) = 20
TEACHING COMMENTARY
To illustrate an instrumental variables calculation, suppose random assignment increases the outcome by twelve units and raises participation from ten to seventy percent. The first stage is sixty percentage points, or zero point six. Dividing twelve by zero point six yields twenty outcome units for people whose participation is induced by the offer. This local interpretation requires a relevant, independent assignment, no people whose participation is reduced by the offer, and an exclusion restriction: assignment affects the outcome only through participation. An offer that also provides cash or encouragement with its own effect can violate exclusion. Therefore, report the assumptions and first stage with the result. Twenty is not automatically the effect for all participants, all eligible people or a future national rollout.

[00:08:39.409 – 00:09:27.963]
Match uncertainty to the assignment structure
ON SCREEN
Cluster where dependence requires it
Report effect and interval together
Distinguish statistical and practical importance

TEACHING COMMENTARY
Uncertainty is about the amount of information in the design, not simply the number of rows in a spreadsheet. If villages are assigned together and their outcomes share shocks, treating every person as an independent assignment can understate uncertainty. Use an inference method appropriate to the assignment and sampling structure, paying particular attention to small numbers of clusters. Report the effect in interpretable units with a confidence interval. Then compare that interval with a prespecified minimum worthwhile effect. An estimate may be distinguishable from zero but still too imprecise to establish a meaningful benefit. Conversely, an interval crossing zero does not prove that the programme has no effect. State which decision-relevant values remain compatible with the evidence and what additional information would help.

[00:09:27.963 – 00:10:23.796]
Plan checks that can change your interpretation
ON SCREEN
Audit missing outcomes and attrition
Prespecify outcomes and subgroup tests
Investigate mechanisms without overclaiming

TEACHING COMMENTARY
A robustness section should investigate concrete threats rather than display many similar regressions. Start with a participant flow table showing assigned, contacted and observed cases by group. If missing outcomes differ, explain possible selection and examine justified sensitivity analyses. A balanced observed sample cannot by itself prove that attrition is harmless. Next, distinguish prespecified outcomes from exploratory findings and account for multiple testing when making families of claims. For subgroup analysis, test whether estimated effects differ; one subgroup being statistically significant and another not significant does not establish a difference. Finally, compare results with implementation evidence. If delivery was weak, the estimate may describe the actual offer under weak implementation rather than the effect of fully delivered training. Preserve that policy distinction.

[00:10:23.796 – 00:11:17.527]
Does a positive estimate clear the decision threshold?
ON SCREEN
Estimated employment effect: 8 percentage points
Standard error: 3 points
Minimum worthwhile effect: 5 points

TEACHING COMMENTARY
Pause and calculate an approximate ninety-five percent confidence interval using the estimate plus or minus one point nine six times its standard error. Keep every quantity in percentage points. Next, answer two separate questions: is zero excluded, and is the minimum worthwhile effect of five points excluded on the low side? Write a recommendation that recognises the difference. Assume the supplied standard error already reflects an appropriate design and a normal approximation is justified. In real work, that assumption requires checking, especially with few clusters. You are practising interpretation rather than selecting the inference method here. Before continuing, identify one additional piece of information a decision maker would need, such as incremental cost, implementation capacity or the consequences of delaying a decision.

[00:11:17.527 – 00:12:08.569]
Evidence of a benefit does not establish sufficient benefit
ON SCREEN
95% interval: 2.12 to 13.88 points
Zero is excluded
Benefits below 5 points remain plausible

TEACHING COMMENTARY
Multiplying three by one point nine six gives five point eight eight percentage points. Subtracting and adding that margin to eight produces an interval from two point one two to thirteen point eight eight. Zero is outside, so this calculation provides evidence of a positive effect under these assumptions. However, the interval includes effects below five points. It therefore does not establish that the programme exceeds the minimum worthwhile effect. A defensible recommendation could support a bounded expansion with further measurement if costs and risks justify it, but that is a decision under uncertainty, not a statistical conclusion. Do not interpret the interval as a ninety-five percent probability that this fixed effect lies inside these observed endpoints; its coverage describes repeated use of the procedure.

[00:12:08.569 – 00:12:59.897]
Translate a valid effect into an incremental cost metric
ON SCREEN
Offer effect: 0.08 × 1,000 = 80
Incremental programme cost: 240,000
Cost per additional employed person: 3,000
240,000 ÷ (0.08 × 1,000) = 3,000
TEACHING COMMENTARY
Suppose the offer effect is eight percentage points across one thousand people offered the programme. Multiplying gives an estimated eighty additional people employed at the measured follow-up date. If incremental programme cost is two hundred and forty thousand monetary units, the ratio is three thousand per additional employed person. Use incremental costs relative to the comparison, with a clearly stated perspective and price year. Do not divide cost by all observed employed participants, because many would have been employed anyway. Employment at one date is also different from sustained employment or lifetime earnings. Explore how the ratio changes across plausible effects and costs, and compare alternatives measured on a consistent basis. A cost-effectiveness ratio informs a decision; it does not replace distributional or implementation judgement.

[00:12:59.897 – 00:13:54.599]
Build a claim that another practitioner can audit
ON SCREEN
Estimand, design and assumptions
Reproducible estimate and sensitivity checks
Decision, limits and next evidence step

TEACHING COMMENTARY
Prepare a short evaluation memo organised around the claim you can defend. State the estimand and describe how assignment or comparison creates the counterfactual. Show participant flow, measurement definitions and the main effect with its uncertainty. Attach reproducible code, a data dictionary and a record of deviations from the analysis plan. Explain the most consequential threats in plain language, including concurrent interventions, spillovers, attrition and limits to transferability. Finish with the decision that follows at current costs and the evidence that would change it. For your own practice, select a recent programme report and rewrite one causal claim at the level its design supports. The objective is disciplined judgement: technically correct calculations joined to an honest explanation of what the evidence identifies.

METHOD REFERENCES
J-PAL — Data analysis
https://www.povertyactionlab.org/resource/data-analysis
Definitions of offer and complier effects; interpretation of instrumental-variable assumptions.

Callaway and Sant’Anna — Introduction to DiD with Multiple Time Periods
https://bcallaway11.github.io/did/articles/multi-period-did.html
Timing-appropriate comparison groups and heterogeneous effects in staggered adoption.

J-PAL — Pre-analysis plans
https://www.povertyactionlab.org/resource/pre-analysis-plans
Prespecification of outcomes and transparent deviations.

J-PAL — Introduction to randomized evaluations
https://www.povertyactionlab.org/resource/introduction-randomized-evaluations
Counterfactual reasoning and assignment-based evaluation design.

Audio credits
https://iderafrica.com/downloads/briefings/audio-credits.txt