Survey sampling and fieldwork quality for credible evidence IDER in-house training materials Hypothetical cases and figures for teaching the methods. Learning objectives Translate a population question into a documented probability sample. Calculate precision allowances, design effects and defensible weights. Implement verification and hand over data for design-aware analysis. Basic proportions, multiplication and familiarity with questionnaire data. Design a survey whose findings can be trusted Sampling, fieldwork and analysis form one system Representativeness requires a documented mechanism All case figures are hypothetical A client asks for a survey of enterprise practices and proposes interviewing whichever owners are easiest to find. Your responsibility is to deliver useful estimates for a population, with uncertainty and limitations that can be defended. This clinic follows a hypothetical survey from the sampling frame through field operations to weighted analysis. We calculate sample requirements, examine clustering, construct weights and distinguish a quality flag from a verified error. Keep a calculator and selection tree available. Junior practitioners should reproduce each calculation; senior practitioners should challenge the assumptions connecting selection to inference. The output is a protocol in which sampling decisions, field procedures and analytical estimators remain consistent from the first sampled unit to the final reported statistic. Define who the survey is intended to represent Target population and reference date Sampling frame and coverage gaps Unit selected versus unit interviewed Specify the target population before discussing sample size. All registered enterprises operating in a district on a reference date is different from all enterprises, including informal firms. A business register may omit recent entrants and include firms that have closed. Compare the frame with available listing information and record known coverage gaps. Next, distinguish the selected unit from the respondent: an enterprise may be selected, while an owner or manager supplies information. Establish which respondent is qualified and how substitutions within the enterprise are handled. More interviews cannot repair a frame systematically excluding the population of interest. A coverage note should therefore accompany the sampling plan, explaining exclusions, updates and the boundaries within which the eventual estimates can reasonably be interpreted. Make every stage of selection reproducible Select areas using recorded probabilities List and select units within areas Preserve identifiers and selection records Imagine selecting areas first, then enterprises from a fresh listing within each area. Store the probability of area selection and the conditional probability of enterprise selection. Their product gives the enterprise's overall selection probability when this design applies. Keep the frame version, randomisation procedure, selected identifiers and any certainty selections. Do not allow field teams to replace an unavailable enterprise with its nearest neighbour merely to meet a quota; that changes inclusion probabilities in ways the original weights do not capture. Instead, use a documented callback and disposition protocol. If a replacement sample is part of the design, specify its probability mechanism in advance. Reproducible selection protects inference and helps distinguish genuine nonresponse from deviations in how interviewers implement the sample. Calculate an initial precision requirement Expected proportion p = 0.50 95% confidence; margin ±5 percentage points 1.96² × 0.50 × 0.50 ÷ 0.05² = 384.16 Initially, assume a large population, simple random sampling and an estimated proportion. If the expected proportion is uncertain, one half gives the largest binomial variance. With a ninety-five percent normal approximation and a desired margin of five percentage points, multiply one point nine six squared by one half times one half, then divide by zero point zero five squared. The result is three hundred and eighty-four point one six, which we round up to three hundred and eighty-five completed interviews. This is a starting requirement, not the final field allocation. Clustering, subgroup reporting and nonresponse can increase it. A substantial sampling fraction may justify a finite population correction, but only when the actual design and population size support that adjustment. Nearby interviews can contain less independent information Average completed cluster size: 20 Intracluster correlation: 0.05 Approximate design effect: 1 + 19 × 0.05 = 1.95 1 + (20 − 1) × 0.05 = 1.95 Cluster sampling can reduce travel costs, but enterprises in the same area may share infrastructure, markets and shocks. Their answers therefore contain overlapping information. For a simplified equal-sized cluster design, an approximate design effect is one plus average completed cluster size minus one, multiplied by the intracluster correlation. With twenty completions per cluster and a correlation of zero point zero five, this becomes one point nine five. This outcome-specific approximation does not automatically capture unequal cluster sizes, weighting or stratification. Obtain plausible inputs from comparable data and test sensitivity before finalising the budget. Adding more independent clusters can improve information differently from adding more interviews within existing clusters. Treat field logistics and statistical precision as a joint design problem rather than separate workstreams. Translate precision into an achievable field sample 385 × 1.95 → 751 completions 751 ÷ 0.80 → 939 initially sampled units Operational allocation: 38 clusters × 25 = 950 Apply the design effect to the rounded simple random requirement. Three hundred and eighty-five times one point nine five equals seven hundred and fifty point seven five, so plan at least seven hundred and fifty-one completions. With an anticipated response rate of eighty percent, divide by zero point eight and round up to nine hundred and thirty-nine initially sampled units. An operational allocation of thirty-eight clusters with twenty-five sampled enterprises each gives nine hundred and fifty selected units and an expected seven hundred and sixty completions. That expectation corresponds to twenty completions per cluster, consistent with our planning input. Response rates and clustering may differ in practice, so monitor them and report achieved precision. Planned numbers are not guarantees of either completion or representativeness. Calculate the information equivalent of a clustered sample Completed interviews: 400 Illustrative design effect: 1.95 Calculate n ÷ design effect and interpret it Pause and calculate the effective sample size for four hundred completed interviews when the design effect is one point nine five. Explain what the result means and what it does not mean. Imagine a manager proposing to delete interviews until the dataset contains only that effective number. Explain why this would be a mistake. The effective size summarises precision relative to a simple random sample for a particular estimator; it is not a list of redundant people. Also write down why the same survey can have different effective sizes for different outcomes. Use the supplied design effect for this exercise, while remembering that actual analysis should use the survey design information to estimate uncertainty rather than rely on this planning shortcut alone. Effective sample size describes precision, not usable records 400 ÷ 1.95 ≈ 205.13 Retain all valid observations Estimate uncertainty using the actual design 400 ÷ 1.95 ≈ 205.13 Four hundred divided by one point nine five is approximately two hundred and five point one three. Under the illustrative assumptions, the estimator has variance comparable to a simple random sample of about two hundred and five observations. All four hundred valid interviews contribute and should be retained. Removing records would generally lose information and may change the estimate. Effective sample size depends on the outcome, estimator, weighting and correlation structure, so it is not a universal property of the file. In final analysis, declare strata, clusters and weights correctly and use a suitable variance estimator. The shortcut is helpful for explaining why a large interview count can provide less precision than expected, but it cannot replace design-based analysis of the realised survey. Invert the probability of entering the sample Area selection probability: 0.20 Conditional enterprise probability: 0.10 Base weight: 1 ÷ (0.20 × 0.10) = 50 Suppose an area has a twenty percent probability of selection, and an enterprise within a selected area has a ten percent conditional probability. Multiplying gives an overall probability of two percent. The inverse is fifty, meaning this sampled enterprise represents fifty enterprises under this design before nonresponse adjustment. Preserve the probability components so the weight can be audited. Do not confuse a large weight with a more truthful response; it reflects representation, not credibility. Check for zero, missing or implausible probabilities and compare weighted totals with the frame. Some designs require more complex inclusion calculations, especially with selection without replacement or unequal probabilities. The key discipline is to derive weights from the actual design rather than assign convenient numbers after seeing the results. Adjust nonresponse using defensible response groups Use measured predictors available for sampled units Adjust within justified cells or models Do not claim that weighting removes all bias A response adjustment tries to account for sampled units whose outcomes are missing. Within a response cell, dividing the base weight by the estimated response rate increases representation for respondents. Its credibility depends on respondents and nonrespondents being comparable for the outcome after conditioning on the adjustment information. Choose cells or a model using relevant variables available for all sampled units, not only respondents. Examine response patterns by size, geography and other pre-existing characteristics, avoiding cells too small for stable estimates. Weighting cannot guarantee removal of bias caused by unmeasured differences. Document callbacks, contact failures, refusals and ineligibility separately because they have different implications. Report the adjustment assumptions and consider sensitivity analyses when plausible nonresponse mechanisms could materially change the substantive conclusion. Weighted estimates can differ from the raw respondent mean 40 respondents: weight 62.5; outcome rate 60% 60 respondents: weight 125; outcome rate 30% Weighted rate = 3,750 ÷ 10,000 = 37.5% Consider two respondent groups. Forty people have weight sixty-two point five and a success rate of sixty percent. Their weighted population is two thousand five hundred, with one thousand five hundred weighted successes. Sixty people have weight one hundred and twenty-five and a success rate of thirty percent. Their weighted population is seven thousand five hundred, with two thousand two hundred and fifty weighted successes. The combined rate is three thousand seven hundred and fifty divided by ten thousand, or thirty-seven point five percent. The unweighted rate is forty-two percent, from forty-two successes among one hundred respondents. The difference reflects unequal representation, not an error. Report the weighted estimator appropriate to the target population and retain enough documentation to explain why those weights are justified. Treat a quality flag as a question to investigate Monitor duration, logic and unusual patterns Verify through authorised checks Correct only with a documented evidence trail A short interview, repeated answer pattern or impossible value should trigger review, not automatic deletion or an accusation against the interviewer. Some short interviews are legitimate because skip logic removes irrelevant sections. Build checks around expected questionnaire paths, range rules and cross-variable consistency. Compare interviewer patterns after accounting for assignment and respondent mix. Use authorised back-checks on selected factual items, with consent and confidentiality protections, to investigate discrepancies. Preserve raw data and log corrections with the reason, evidence, reviewer and date. Do not overwrite a surprising but verified response merely because it weakens a preferred finding. A good quality system separates detection, verification and resolution, giving field teams timely feedback while maintaining a defensible audit trail from the original observation to the analysis file. Combine selection and response adjustment Area probability: 0.20; enterprise probability: 0.10 Response rate within an appropriate cell: 0.80 Calculate the adjusted weight and state its assumption Pause and calculate the base weight, then the response-adjusted weight for these probabilities. Explain the role of each factor rather than only reporting a number. State the key assumption required for the response adjustment to reduce bias. Next, consider a fieldworker who replaces an unavailable selected enterprise with a nearby willing owner. Would the calculated weight automatically become valid for that replacement? Explain why not. Your answer should connect the arithmetic to the actual recruitment process. Finally, write one practical action a supervisor can take when response falls below plan, without abandoning the probability sample. Examples might concern contact attempts, scheduling or a previously specified design change. Maintain the link between selection, fieldwork and inference under operational pressure. The adjusted weight is 62.5, conditional on its assumptions Base weight: 1 ÷ 0.02 = 50 Response adjustment: 50 ÷ 0.80 = 62.5 Convenience substitution breaks the documented selection link 1 ÷ (0.20 × 0.10 × 0.80) = 62.5 The area and conditional enterprise probabilities multiply to zero point zero two, giving a base weight of fifty. Dividing by the cell response rate of zero point eight gives sixty-two point five. This adjustment assumes the response model or cell captures the relevant differences between respondents and nonrespondents for the intended inference. It does not make unmeasured selection disappear. A nearby replacement does not automatically share the selected enterprise's inclusion probability, so transferring the weight is unjustified. A supervisor should follow documented callbacks, vary contact times where appropriate and record outcomes accurately. If a design modification becomes necessary, obtain the required methodological review and document its probability implications. Meeting an interview quota is not a sufficient reason to sacrifice a reproducible sampling mechanism. Deliver design information with the cleaned data Strata, cluster IDs and final weights Disposition and correction records Domain estimates with design-aware uncertainty A spreadsheet is not a complete survey handover. Supply a data dictionary, questionnaire versions, sampling frame notes, selection probabilities, cluster and stratum identifiers, and the final weighting procedure. Include a disposition summary explaining response and eligibility, along with a correction log that preserves original values. Provide analysis code that calculates estimates and their uncertainty using the actual design. For subgroup reporting, use a suitable domain estimation approach rather than casually discarding observations needed for variance estimation. Check weighted totals and identify unstable estimates or sparse domains before publication. Keep identifiers needed for design analysis while removing or protecting direct personal identifiers according to the agreed data arrangements. This package allows a later analyst to reproduce results and judge the limits of what the survey represents. Build a survey protocol that stays coherent in the field Population, frame and probability sample Field procedures aligned with selection Weighted estimates, uncertainty and limitations Prepare a single protocol linking the target population to the final estimator. Start with the frame and its coverage limits, then draw the selection stages and calculate inclusion probabilities. Show the precision assumptions, cluster allocation and response allowance separately. Write callback, consent and quality-review procedures that protect the selected sample. Before launch, pilot the questionnaire and data checks using realistic cases, including nonresponse and unusual but valid answers. After fieldwork, compare planned and achieved selection, response and precision, explaining every material deviation. Your final report should present weighted estimates, appropriate uncertainty and limitations without promising that a large sample solves all bias. Professional survey practice depends on maintaining this chain of evidence from design through collection to interpretation. Check your understanding 1. Under the illustrative simple random planning assumptions, what completed sample follows from 384.16? A. 384 B. 385 C. 751 D. 950 2. With 20 completed interviews per cluster and ICC 0.05, what is the approximate design effect? A. 1.05 B. 1.50 C. 1.95 D. 20.05 3. What does an effective sample size of about 205 from 400 interviews imply? A. Delete 195 records B. Only 205 people responded C. Approximate precision comparable to an SRS of 205 for the specified estimator D. All outcomes have the same effective size 4. Selection probabilities are 0.20 and 0.10, and cell response is 0.80. What is the adjusted weight? A. 12.5 B. 40 C. 50 D. 62.5 5. Forty respondents have weight 62.5 and success 60%; sixty have weight 125 and success 30%. What is the weighted success rate? A. 42% B. 45% C. 37.5% D. 30% 6. An interview is unusually short. What is the appropriate first response? A. Delete it automatically B. Accuse the interviewer C. Investigate skip logic and other evidence through the quality protocol D. Replace all its answers with averages Applied task 1 Prepare the hypothetical survey allocation from SRS requirement 385, design effect 1.95 and response 80%. Explain why 38 clusters of 25 selected units is coherent. Applied task 2 Write a field and analysis protocol for selection probabilities 0.20 and 0.10 with response 0.80. Include nonresponse and unusual interviews. Answers and explanations 1. B. Round the required completed sample up to 385 before applying the additional planning adjustments used in the case. (Slide 4) 2. C. 1 + (20 − 1) × 0.05 = 1.95 under the simplified equal-sized cluster approximation. (Slide 5) 3. C. It summarises relative variance, not usable record count; all valid interviews remain in the analysis. (Slide 8) 4. D. The base weight is 1/(0.20 × 0.10) = 50; adjusting gives 50/0.80 = 62.5, conditional on appropriate response assumptions. (Slide 14) 5. C. Weighted successes are 1,500 + 2,250 = 3,750; weighted population is 2,500 + 7,500 = 10,000. (Slide 11) 6. C. A flag is a reason for verification; a short interview can be valid. Corrections require a documented evidence trail. (Slide 12) Applied task 1 The adjusted completion requirement is ceil(385 × 1.95) = 751. The initial selection requirement is ceil(751/0.80) = 939. Selecting 38 × 25 = 950 yields 760 expected responses, or 20 per cluster, consistent with the design-effect assumption. Achieved response and precision must still be checked. Correct rounding and distinction between selected and completed units. Allocation is linked to expected completed cluster size. States assumptions and avoids guaranteeing achieved precision. Applied task 2 Record both probability stages; base weight is 50 and adjusted weight is 62.5 under defensible response-cell assumptions. Use callbacks and dispositions rather than convenience replacements. Investigate flags with authorised verification, preserve raw data and correction logs, and deliver strata, clusters, weights and design-aware analysis code. Correct weights with an explicit response assumption. Field procedures preserve selection and verify rather than automatically delete. Handover contains design information and reproducible analysis. References United Nations Statistics Division — Designing Household Survey Samples: Practical Guidelines https://unstats.un.org/unsd/demographic/sources/surveys/Handbook23June05.pdf Probability sampling, multistage selection, design effects, weighting and survey documentation. J-PAL — Sampling https://www.povertyactionlab.org/resource/sampling Population, frame and sampling choices. J-PAL — Data quality checks https://www.povertyactionlab.org/resource/data-quality-checks Operational checks and verification of field data. J-PAL — Power calculations https://www.povertyactionlab.org/resource/power-calculations Relationship between sample design, clustering and statistical information; precision examples here are original calculations, not treatment-effect power calculations.