What you will be able to do
- Define an estimand and defend a counterfactual comparison.
- Calculate and interpret DiD and complier effects with their assumptions.
- Connect uncertainty and incremental costs to a defensible decision.
Before you begin: Basic percentages, averages and familiarity with programme monitoring.
Write the estimand before choosing the estimator
Start by writing a complete sentence describing the effect you want to estimate. For example, what is the effect of offering training to eligible applicants on employment twelve months later? That sentence fixes the population, intervention, outcome and horizon. It also identifies the offer as the policy lever. Comparing people who attended with people who did not attend answers a different question because attendance may reflect motivation or available time. Create an estimand register before analysing outcomes. Include the unit of assignment, unit of observation and the population to which you expect results to apply. If these definitions change later, preserve the original version and explain why. Clear definitions prevent a technically sophisticated model from answering an irrelevant question.
Difference in differences removes a shared change
Consider average monthly profit measured in the same monetary units for two groups. Treated businesses increase from one hundred to one hundred and forty. Comparison businesses increase from eighty to one hundred. The treated change is forty, but twenty units of improvement also occurred without the programme. Subtracting the comparison change gives an estimated effect of twenty. Another way to see this is to add the comparison change to the treated baseline: one hundred plus twenty predicts a counterfactual profit of one hundred and twenty. The observed treated outcome exceeds that prediction by twenty. The arithmetic is easy; credibility depends on whether untreated trends would have been parallel, together with stable measurement, appropriate composition and no differential concurrent shock.
A participation effect requires additional assumptions
To illustrate an instrumental variables calculation, suppose random assignment increases the outcome by twelve units and raises participation from ten to seventy percent. The first stage is sixty percentage points, or zero point six. Dividing twelve by zero point six yields twenty outcome units for people whose participation is induced by the offer. This local interpretation requires a relevant, independent assignment, no people whose participation is reduced by the offer, and an exclusion restriction: assignment affects the outcome only through participation. An offer that also provides cash or encouragement with its own effect can violate exclusion. Therefore, report the assumptions and first stage with the result. Twenty is not automatically the effect for all participants, all eligible people or a future national rollout.
Translate a valid effect into an incremental cost metric
Suppose the offer effect is eight percentage points across one thousand people offered the programme. Multiplying gives an estimated eighty additional people employed at the measured follow-up date. If incremental programme cost is two hundred and forty thousand monetary units, the ratio is three thousand per additional employed person. Use incremental costs relative to the comparison, with a clearly stated perspective and price year. Do not divide cost by all observed employed participants, because many would have been employed anyway. Employment at one date is also different from sustained employment or lifetime earnings. Explore how the ratio changes across plausible effects and costs, and compare alternatives measured on a consistent basis. A cost-effectiveness ratio informs a decision; it does not replace distributional or implementation judgement.
Apply the method
Work through each task before opening the model answer. Compare your reasoning with the review criteria. Written answers stay on this page and clear when you leave or reload it.
Applied task 1
Show model answer and review criteria
A strong design defines eligible applicants, the effect of an offer on employment at twelve months, and a feasible allocation or justified comparison rule. It measures baseline and follow-up consistently, records concurrent support, preserves assignment, and plans attrition and spillover checks. Alternative designs are acceptable when their identifying assumptions are explicit.
- Estimand fixes population, lever, outcome and horizon.
- Counterfactual and assignment or selection mechanism are explained.
- Threats have specific measurement or design responses.
Applied task 2
Show model answer and review criteria
The approximate 95% interval is 2.12–13.88 points; the point estimate implies 80 additional employed people and 3,000 per additional person. Evidence supports a positive effect but does not establish exceeding 5 points. Recommend an action conditional on costs, risk and implementation capacity, and identify what further evidence would change it.
- Correct interval and additional-outcome calculation.
- Distinguishes positive evidence from exceeding the threshold.
- Recommendation acknowledges uncertainty and incremental cost.
Method references
- J-PAL — Data analysis
Definitions of offer and complier effects; interpretation of instrumental-variable assumptions.
- Callaway and Sant’Anna — Introduction to DiD with Multiple Time Periods
Timing-appropriate comparison groups and heterogeneous effects in staggered adoption.
- J-PAL — Pre-analysis plans
Prespecification of outcomes and transparent deviations.
- J-PAL — Introduction to randomized evaluations
Counterfactual reasoning and assignment-based evaluation design.