Introduction

Census testing, including pre-tests, pilot and post-pilot testing, is a major activity conducted during the pre-enumeration phase. All aspects of census preparedness need to be comprehensively tested, ranging from administrative, logistical, technical, and technological infrastructure. In addition, the interface between enumeration, processing, analysis, and dissemination systems needs to be tested to resolve any outstanding problems.

In a digital census, various tests must be conducted before and after the pilot census. It is essential for countries to prioritise undertaking adequate tests of the various aspects of a census plan. The initial tests carried out during census preparations include testing of tools and instruments often referred to as census pretests and these can be several and on a small scale for different purposes. For instance, a good questionnaire test should iteratively first evaluate the performance of a questionnaire before changes are made; then make any changes to improve its performance; and finally, evaluate the questionnaire after changes are made to find out if its performance has improved.

The advantage of carrying out a pilot census is that it provides an opportunity to perfect the census preparedness as a whole, and address any issues prior to the census, as well as revise the costing estimates. To obtain accurate costing estimates from the pilot, all the aspects of the census need to be finalised at this stage after testing. Given the significantly smaller volume of data and field staff being used for the pilot, bear in mind when evaluating the pilot and noting small problems, that although these may be dealt with quickly in the pilot, they may be significantly more troublesome in the main event. Additionally, there should be plans after the pilot and before the main operations to stress test the information and communication technology (ICT) systems with realistic volumes of (artificial) data.

Census testing and pilots during the non-digital vs digital era

During the non-digital census era (paper questionnaire), testing focused mainly on testing the quality of the Enumeration Area (EA) maps, assessment of the enumerator’s workload and enumeration duration, overall administrative and logistics arrangements, terrain, and understanding of the questionnaire. 

In a digital census there are many additional areas to be tested due to the various interfaces between systems and people. These include testing of the data collection application, its design and architecture, the internet, the equipment (handheld devices, servers etc.), the data transmission mechanisms; the readiness of the census data centre, the data integrity and data security, the data quality monitoring system/dashboard including quality of the EA maps as well as the underlying circumstances necessary to avoid equipment malfunctioning or unexpected problems in programming processes. It is also important to stress test digital systems such as e-recruitment systems and data transmission systems to ensure that they can handle the large volumes of data traffic that will be expected during the census.

All digital components such as e-recruitment platforms, enumerator payment systems, CAPI application, data management system, monitoring dashboard and so on, should be thoroughly tested prior to the census pilot. After each test, if any changes are made, this necessitates testing again. Sufficient time and budget should be reserved for this retesting process. By the time of the pilot, all systems should be ready as if this were the census. The pilot is thus an opportunity to test how everything works together and at scale. The pilot will undoubtedly highlight some issues which can be addressed prior to the census. Components that are changed after the pilot need to be tested again to ensure that everything is working according to plan by the time of the census.

Considerations for a digital pilot census

The main objective of a digital pilot census is to test all the aspects of the census including the ICT infrastructure for the census. Therefore, the following key considerations should be made for a successful pilot census.

• The census project schedule should include several field test exercises to be undertaken prior to the formal pilot census using the selected or developed devices and systems to test the reliability in the different circumstances presented in the field. As well as allowing adequate time to make any necessary improvements prior to deployment of the technologies in the field.
• A pilot census should set clear objectives for the main areas to be tested, including the relevant evaluation criteria, and identify data/information requirements to inform the assessment. This may impact how your pilot is designed. Evaluation of the pilot should be done in a timely manner as the results may necessitate an overhaul of the infrastructure and significant revisions to the population and housing census (PHC) business process. This overhaul will call for pretesting of the revised approach at an extra cost.
• When developing a pilot census plan, consideration should be made on how the pilot sample is constituted to ensure that the selected areas provide a sufficient test of the setout objectives. Therefore, the sample should be purposefully selected to include rural/urban, areas with poor or no internet connectivity, other challenging areas, special and hard-to-count populations within a country’s context.
• A pilot census should be conducted in conditions that are as close to the conditions that would be present during the actual enumeration as possible. For this reason, it is often recommended to be undertaken exactly one year before the planned census enumeration to conform to the expected seasonal patterns of climate and activity.
• The questionnaire design should be final with no or minimal changes expected, at the time of the pilot census. This includes translations into other languages. Therefore, the questionnaire should be extensively tested prior to the pilot. Any changes made after the pilot should be tested.
• Unanticipated circumstances often arise during pilot census enumeration. Priority should be given to each detail for debriefing and making informed reviews of the various components being tested. Therefore, each issue identified during the pilot should be documented in a lessons-learnt document to be considered and resolved before the census. A checklist of the tests scheduled, and the recommendation should be tracked, and responsible officers assigned to fast track. Any omission or disregard of issues that look minor during the tests and pilot, may have significant effect during the actual census enumeration. Debriefing sessions should be properly structured, and documentation carried out systematically.
• After the pilot census, the implemented recommendations resulting from the pilot census process should be tested. It is desirable that these tests target all stages of the census, including enumeration, data processing and evaluation of results to give a clear picture of the adequacy of the entire census plan and organisation.