WEBVTT NOTE IDER in-house training materials. Captions follow measured phrase audio timings. Hypothetical worked cases; figures are instructional examples. 00:00:00.000 --> 00:00:07.621 A client asks for a survey of enterprise practices and proposes interviewing whichever owners are easiest to 00:00:07.621 --> 00:00:08.069 find. 00:00:08.189 --> 00:00:14.778 Your responsibility is to deliver useful estimates for a population, with uncertainty and limitations that 00:00:14.778 --> 00:00:16.095 can be defended. 00:00:16.215 --> 00:00:22.136 This clinic follows a hypothetical survey from the sampling frame through field operations to weighted 00:00:22.136 --> 00:00:22.531 analysis. 00:00:22.651 --> 00:00:29.471 We calculate sample requirements, examine clustering, construct weights and distinguish a quality flag from 00:00:29.471 --> 00:00:34.342 a verified error. Keep a calculator and selection tree available. 00:00:34.462 --> 00:00:40.872 Junior practitioners should reproduce each calculation; senior practitioners should challenge the 00:00:40.872 --> 00:00:43.785 assumptions connecting selection to inference. 00:00:43.905 --> 00:00:50.061 The output is a protocol in which sampling decisions, field procedures and analytical estimators remain 00:00:50.061 --> 00:00:54.575 consistent from the first sampled unit to the final reported statistic. 00:00:55.245 --> 00:01:02.210 Specify the target population before discussing sample size. All registered enterprises operating in a 00:01:02.210 --> 00:01:08.678 district on a reference date is different from all enterprises, including informal firms. 00:01:08.798 --> 00:01:15.871 A business register may omit recent entrants and include firms that have closed. Compare the frame with 00:01:15.871 --> 00:01:19.200 available listing information and record known coverage gaps. 00:01:19.320 --> 00:01:26.269 Next, distinguish the selected unit from the respondent: an enterprise may be selected, while an 00:01:26.269 --> 00:01:28.585 owner or manager supplies information. 00:01:28.705 --> 00:01:35.440 Establish which respondent is qualified and how substitutions within the enterprise are handled. More 00:01:35.440 --> 00:01:40.733 interviews cannot repair a frame systematically excluding the population of interest. 00:01:40.853 --> 00:01:47.029 A coverage note should therefore accompany the sampling plan, explaining exclusions, updates and the 00:01:47.029 --> 00:01:51.441 boundaries within which the eventual estimates can reasonably be interpreted. 00:01:52.111 --> 00:01:59.691 Imagine selecting areas first, then enterprises from a fresh listing within each area. Store the probability 00:01:59.691 --> 00:02:04.429 of area selection and the conditional probability of enterprise selection. 00:02:04.549 --> 00:02:12.400 Their product gives the enterprise's overall selection probability when this design applies. Keep the frame 00:02:12.400 --> 00:02:17.111 version, randomisation procedure, selected identifiers and any certainty selections. 00:02:17.231 --> 00:02:23.979 Do not allow field teams to replace an unavailable enterprise with its nearest neighbour merely to meet a 00:02:23.979 --> 00:02:28.853 quota; that changes inclusion probabilities in ways the original weights do not capture. 00:02:28.973 --> 00:02:36.726 Instead, use a documented callback and disposition protocol. If a replacement sample is part of the 00:02:36.726 --> 00:02:40.118 design, specify its probability mechanism in advance. 00:02:40.238 --> 00:02:46.941 Reproducible selection protects inference and helps distinguish genuine nonresponse from deviations in how 00:02:46.941 --> 00:02:49.004 interviewers implement the sample. 00:02:49.674 --> 00:02:57.887 Initially, assume a large population, simple random sampling and an estimated proportion. If the expected 00:02:57.887 --> 00:03:03.362 proportion is uncertain, one half gives the largest binomial variance. 00:03:03.482 --> 00:03:10.222 With a ninety-five percent normal approximation and a desired margin of five percentage points, multiply one 00:03:10.222 --> 00:03:16.961 point nine six squared by one half times one half, then divide by zero point zero 00:03:17.081 --> 00:03:25.177 five squared. The result is three hundred and eighty-four point one six, which we round up to three 00:03:25.177 --> 00:03:27.426 hundred and eighty-five completed interviews. 00:03:27.546 --> 00:03:34.463 This is a starting requirement, not the final field allocation. Clustering, subgroup reporting and 00:03:34.463 --> 00:03:36.439 nonresponse can increase it. 00:03:36.559 --> 00:03:43.854 A substantial sampling fraction may justify a finite population correction, but only when the actual design 00:03:43.854 --> 00:03:46.590 and population size support that adjustment. 00:03:47.260 --> 00:03:54.531 Cluster sampling can reduce travel costs, but enterprises in the same area may share infrastructure, 00:03:54.531 --> 00:03:58.893 markets and shocks. Their answers therefore contain overlapping information. 00:03:59.013 --> 00:04:06.287 For a simplified equal-sized cluster design, an approximate design effect is one plus average 00:04:06.287 --> 00:04:11.482 completed cluster size minus one, multiplied by the intracluster correlation. 00:04:11.602 --> 00:04:21.104 With twenty completions per cluster and a correlation of zero point zero five, this becomes one point nine 00:04:21.104 --> 00:04:27.439 five. This outcome-specific approximation does not automatically capture unequal cluster sizes, weighting 00:04:27.439 --> 00:04:28.495 or stratification. 00:04:28.615 --> 00:04:36.078 Obtain plausible inputs from comparable data and test sensitivity before finalising the budget. Adding more 00:04:36.078 --> 00:04:41.552 independent clusters can improve information differently from adding more interviews within 00:04:41.552 --> 00:04:42.547 existing clusters. 00:04:42.667 --> 00:04:49.993 Treat field logistics and statistical precision as a joint design problem rather than separate workstreams. 00:04:50.663 --> 00:04:54.099 Apply the design effect to the rounded simple random requirement. 00:04:54.219 --> 00:05:01.461 Three hundred and eighty-five times one point nine five equals seven hundred and fifty point seven five, 00:05:01.461 --> 00:05:05.295 so plan at least seven hundred and fifty-one completions. 00:05:05.415 --> 00:05:12.964 With an anticipated response rate of eighty percent, divide by zero point eight and round up to nine 00:05:12.964 --> 00:05:15.481 hundred and thirty-nine initially sampled units. 00:05:15.601 --> 00:05:22.074 An operational allocation of thirty-eight clusters with twenty-five sampled enterprises each gives nine 00:05:22.074 --> 00:05:28.546 hundred and fifty selected units and an expected seven hundred and sixty completions. 00:05:28.666 --> 00:05:34.906 That expectation corresponds to twenty completions per cluster, consistent with our planning input. Response 00:05:34.906 --> 00:05:41.147 rates and clustering may differ in practice, so monitor them and report achieved precision. 00:05:41.267 --> 00:05:45.795 Planned numbers are not guarantees of either completion or representativeness. 00:05:46.465 --> 00:05:52.355 Pause and calculate the effective sample size for four hundred completed interviews when the design effect is 00:05:52.355 --> 00:05:57.552 one point nine five. Explain what the result means and what it does not mean. 00:05:57.672 --> 00:06:03.047 Imagine a manager proposing to delete interviews until the dataset contains only that effective number. 00:06:03.047 --> 00:06:05.555 Explain why this would be a mistake. 00:06:05.675 --> 00:06:12.060 The effective size summarises precision relative to a simple random sample for a particular estimator; it is 00:06:12.060 --> 00:06:14.313 not a list of redundant people. 00:06:14.433 --> 00:06:19.971 Also write down why the same survey can have different effective sizes for different outcomes. 00:06:20.091 --> 00:06:26.828 Use the supplied design effect for this exercise, while remembering that actual analysis should use the 00:06:26.828 --> 00:06:32.723 survey design information to estimate uncertainty rather than rely on this planning shortcut alone. 00:06:33.393 --> 00:06:39.198 Four hundred divided by one point nine five is approximately two hundred and five point one three. 00:06:39.318 --> 00:06:46.191 Under the illustrative assumptions, the estimator has variance comparable to a simple random sample of about 00:06:46.191 --> 00:06:52.634 two hundred and five observations. All four hundred valid interviews contribute and should be retained. 00:06:52.754 --> 00:06:57.247 Removing records would generally lose information and may change the estimate. 00:06:57.367 --> 00:07:03.559 Effective sample size depends on the outcome, estimator, weighting and correlation structure, so it 00:07:03.559 --> 00:07:07.097 is not a universal property of the file. 00:07:07.217 --> 00:07:15.008 In final analysis, declare strata, clusters and weights correctly and use a suitable variance 00:07:15.008 --> 00:07:15.564 estimator. 00:07:15.684 --> 00:07:22.000 The shortcut is helpful for explaining why a large interview count can provide less precision than 00:07:22.000 --> 00:07:26.342 expected, but it cannot replace design-based analysis of the realised survey. 00:07:27.012 --> 00:07:33.907 Suppose an area has a twenty percent probability of selection, and an enterprise within a selected area 00:07:33.907 --> 00:07:39.586 has a ten percent conditional probability. Multiplying gives an overall probability of two percent. 00:07:39.706 --> 00:07:46.633 The inverse is fifty, meaning this sampled enterprise represents fifty enterprises under this design before 00:07:46.633 --> 00:07:52.175 nonresponse adjustment. Preserve the probability components so the weight can be audited. 00:07:52.295 --> 00:07:58.827 Do not confuse a large weight with a more truthful response; it reflects representation, not credibility. 00:07:58.827 --> 00:08:04.543 Check for zero, missing or implausible probabilities and compare weighted totals with the frame. 00:08:04.663 --> 00:08:10.776 Some designs require more complex inclusion calculations, especially with selection without 00:08:10.776 --> 00:08:12.999 replacement or unequal probabilities. 00:08:13.119 --> 00:08:19.537 The key discipline is to derive weights from the actual design rather than assign convenient numbers 00:08:19.537 --> 00:08:21.142 after seeing the results. 00:08:21.812 --> 00:08:28.450 A response adjustment tries to account for sampled units whose outcomes are missing. Within a response 00:08:28.450 --> 00:08:33.843 cell, dividing the base weight by the estimated response rate increases representation for 00:08:33.843 --> 00:08:34.258 respondents. 00:08:34.378 --> 00:08:40.071 Its credibility depends on respondents and nonrespondents being comparable for the outcome after 00:08:40.071 --> 00:08:42.261 conditioning on the adjustment information. 00:08:42.381 --> 00:08:48.848 Choose cells or a model using relevant variables available for all sampled units, not only respondents. 00:08:48.968 --> 00:08:56.432 Examine response patterns by size, geography and other pre-existing characteristics, avoiding cells too small 00:08:56.432 --> 00:09:03.364 for stable estimates. Weighting cannot guarantee removal of bias caused by unmeasured differences. 00:09:03.484 --> 00:09:10.468 Document callbacks, contact failures, refusals and ineligibility separately because they have different 00:09:10.468 --> 00:09:16.870 implications. Report the adjustment assumptions and consider sensitivity analyses when plausible 00:09:16.870 --> 00:09:21.526 nonresponse mechanisms could materially change the substantive conclusion. 00:09:22.196 --> 00:09:28.914 Consider two respondent groups. Forty people have weight sixty-two point five and a success rate of 00:09:28.914 --> 00:09:29.754 sixty percent. 00:09:29.874 --> 00:09:35.650 Their weighted population is two thousand five hundred, with one thousand five hundred weighted 00:09:35.650 --> 00:09:42.250 successes. Sixty people have weight one hundred and twenty-five and a success rate of thirty percent. 00:09:42.370 --> 00:09:48.568 Their weighted population is seven thousand five hundred, with two thousand two hundred and fifty 00:09:48.568 --> 00:09:49.394 weighted successes. 00:09:49.514 --> 00:09:57.292 The combined rate is three thousand seven hundred and fifty divided by ten thousand, or thirty-seven point 00:09:57.292 --> 00:10:03.697 five percent. The unweighted rate is forty-two percent, from forty-two successes among one hundred 00:10:03.697 --> 00:10:04.154 respondents. 00:10:04.274 --> 00:10:10.979 The difference reflects unequal representation, not an error. Report the weighted estimator appropriate to 00:10:10.979 --> 00:10:17.684 the target population and retain enough documentation to explain why those weights are justified. 00:10:18.354 --> 00:10:25.038 A short interview, repeated answer pattern or impossible value should trigger review, not automatic 00:10:25.038 --> 00:10:31.723 deletion or an accusation against the interviewer. Some short interviews are legitimate because skip 00:10:31.723 --> 00:10:33.633 logic removes irrelevant sections. 00:10:33.753 --> 00:10:40.228 Build checks around expected questionnaire paths, range rules and cross-variable consistency. Compare 00:10:40.228 --> 00:10:45.084 interviewer patterns after accounting for assignment and respondent mix. 00:10:45.204 --> 00:10:51.996 Use authorised back-checks on selected factual items, with consent and confidentiality protections, to 00:10:51.996 --> 00:10:59.310 investigate discrepancies. Preserve raw data and log corrections with the reason, evidence, reviewer and 00:10:59.310 --> 00:10:59.833 date. 00:10:59.953 --> 00:11:06.559 Do not overwrite a surprising but verified response merely because it weakens a preferred finding. 00:11:06.679 --> 00:11:13.201 A good quality system separates detection, verification and resolution, giving field teams timely 00:11:13.201 --> 00:11:20.727 feedback while maintaining a defensible audit trail from the original observation to the analysis file. 00:11:21.397 --> 00:11:26.661 Pause and calculate the base weight, then the response-adjusted weight for these probabilities. 00:11:26.661 --> 00:11:31.521 Explain the role of each factor rather than only reporting a number. 00:11:31.641 --> 00:11:37.755 State the key assumption required for the response adjustment to reduce bias. Next, consider a 00:11:37.755 --> 00:11:42.647 fieldworker who replaces an unavailable selected enterprise with a nearby willing owner. 00:11:42.767 --> 00:11:49.058 Would the calculated weight automatically become valid for that replacement? Explain why not. Your answer 00:11:49.058 --> 00:11:52.833 should connect the arithmetic to the actual recruitment process. 00:11:52.953 --> 00:12:00.379 Finally, write one practical action a supervisor can take when response falls below plan, without 00:12:00.379 --> 00:12:06.816 abandoning the probability sample. Examples might concern contact attempts, scheduling or a previously 00:12:06.816 --> 00:12:08.301 specified design change. 00:12:08.421 --> 00:12:14.064 Maintain the link between selection, fieldwork and inference under operational pressure. 00:12:14.734 --> 00:12:21.520 The area and conditional enterprise probabilities multiply to zero point zero two, giving a base weight 00:12:21.520 --> 00:12:28.306 of fifty. Dividing by the cell response rate of zero point eight gives sixty-two point five. 00:12:28.426 --> 00:12:34.602 This adjustment assumes the response model or cell captures the relevant differences between respondents 00:12:34.602 --> 00:12:40.337 and nonrespondents for the intended inference. It does not make unmeasured selection disappear. 00:12:40.457 --> 00:12:47.342 A nearby replacement does not automatically share the selected enterprise's inclusion probability, so 00:12:47.342 --> 00:12:49.989 transferring the weight is unjustified. 00:12:50.109 --> 00:12:57.486 A supervisor should follow documented callbacks, vary contact times where appropriate and record outcomes 00:12:57.486 --> 00:13:03.810 accurately. If a design modification becomes necessary, obtain the required methodological review 00:13:03.810 --> 00:13:06.444 and document its probability implications. 00:13:06.564 --> 00:13:12.706 Meeting an interview quota is not a sufficient reason to sacrifice a reproducible sampling mechanism. 00:13:13.376 --> 00:13:20.667 A spreadsheet is not a complete survey handover. Supply a data dictionary, questionnaire versions, 00:13:20.667 --> 00:13:27.437 sampling frame notes, selection probabilities, cluster and stratum identifiers, and the final weighting 00:13:27.437 --> 00:13:27.958 procedure. 00:13:28.078 --> 00:13:35.214 Include a disposition summary explaining response and eligibility, along with a correction log that 00:13:35.214 --> 00:13:42.349 preserves original values. Provide analysis code that calculates estimates and their uncertainty using the 00:13:42.349 --> 00:13:43.369 actual design. 00:13:43.489 --> 00:13:50.160 For subgroup reporting, use a suitable domain estimation approach rather than casually discarding 00:13:50.160 --> 00:13:56.831 observations needed for variance estimation. Check weighted totals and identify unstable estimates or 00:13:56.831 --> 00:13:58.883 sparse domains before publication. 00:13:59.003 --> 00:14:04.985 Keep identifiers needed for design analysis while removing or protecting direct personal identifiers 00:14:04.985 --> 00:14:07.746 according to the agreed data arrangements. 00:14:07.866 --> 00:14:13.666 This package allows a later analyst to reproduce results and judge the limits of what the survey 00:14:13.666 --> 00:14:14.007 represents. 00:14:14.677 --> 00:14:20.827 Prepare a single protocol linking the target population to the final estimator. Start with the 00:14:20.827 --> 00:14:26.566 frame and its coverage limits, then draw the selection stages and calculate inclusion probabilities. 00:14:26.686 --> 00:14:33.744 Show the precision assumptions, cluster allocation and response allowance separately. Write callback, consent 00:14:33.744 --> 00:14:38.087 and quality-review procedures that protect the selected sample. 00:14:38.207 --> 00:14:45.577 Before launch, pilot the questionnaire and data checks using realistic cases, including nonresponse and 00:14:45.577 --> 00:14:52.421 unusual but valid answers. After fieldwork, compare planned and achieved selection, response and 00:14:52.421 --> 00:14:55.053 precision, explaining every material deviation. 00:14:55.173 --> 00:15:01.150 Your final report should present weighted estimates, appropriate uncertainty and limitations without 00:15:01.150 --> 00:15:05.134 promising that a large sample solves all bias. 00:15:05.254 --> 00:15:11.116 Professional survey practice depends on maintaining this chain of evidence from design through collection 00:15:11.116 --> 00:15:11.953 to interpretation.