This is a guest blog by Daniel W. Smith, a Water & Sanitation Advisor at the Center for Water Security, Sanitation, and Hygiene at USAID in Washington, DC.
Photo: A handpump mechanic performs preventive maintenance in Uganda
(Photo: Daniel W. Smith)
If you measure something, how do you know that someone else would get the same result? This is a fundamental question in many fields including medicine and psychology, but it is rarely considered in rural water supply.
This problem became painfully apparent during a recent study of professionalizing handpump maintenance in Uganda conducted by the Program for Water, Health, and Development at the Stanford Woods Institute for the Environment and International Lifeline Fund. Our data collection team had a seemingly straightforward instruction: Count a handpump as functional if it provides water. But different data collectors interpreted the instruction differently. Some would count a handpump as functional even if it took a long time to get a little water. Others counted handpumps in a similar condition as nonfunctional. We needed a clearer, more reliable procedure to ensure that handpump functionality measured by different people would be comparable.
To do so, we conducted a study in which teams of data collectors repeatedly tested the functionality of the same handpumps. And we didn’t just measure functionality as providing water or not. We applied three field tests developed by International Lifeline Fund that we hoped would provide more reliable and nuanced information. The three tests were:
- Pump capacity, which measures how much water comes out with each pump stroke;
- 10-minute leakage rate, which measures how fast water leaks from the pumping column; and
- Flowrate.
These tests are quick to complete, so we were able to repeat them hundreds of times. Using statistical techniques, we then calculated how close the different data collectors’ results were to each other, which is known as inter-rater reliability. This is probably the first time inter-rater reliability has been calculated for anything related to rural water supply, even though most monitoring and research needs more than one person to collect data.
What we found was at once reassuring and concerning. The good news: Different people’s 10-minute leak tests were nearly identical, and their pump capacity tests were reasonably comparable. The bad news: Different people got very different flowrates despite our best efforts to standardize the procedure.
Our data led to three practical outcomes:
- We created a simple procedure based on the 10-minute leakage rate and pump capacity tests that anyone can use to reliably measure handpump functionality (see figure below). This procedure could be incorporated into service delivery monitoring and research to enhance the objectivity, utility, and comparability of global handpump functionality data.
- Functionality measured this way could predict handpump breakdown. Future studies could test this hypothesis, which would be useful for operation and maintenance initiatives.
- Even though handpump flowrate is commonly measured as a dimension of handpump functionality, results from different people can’t be compared.
We hope that our findings contribute to the current debate about how to measure water point functionality in reliable and cost-effective ways. We welcome feedback from the RWSN community.
Inter- and intra-rater reliability of handpump functionality field tests. Science of The Total Environment. Volume 869, 161616. https://doi.org/10.1016/j.scitotenv.2023.161616).
Disclaimer: The opinions and views expressed are those of the author and not necessarily those of USAID.
About the author:
Daniel W. Smith, PhD, MSc is a Water & Sanitation Advisor at the Center for Water Security, Sanitation, and Hygiene at USAID, where he supports USAID country WASH programs, strategy development, and implementation research. He conducted the work featured in this blog at the Stanford Woods Institute for the Environment before joining USAID.
Featured research:
Daniel W. Smith, Samantha Wind, Igor A. Markov, Stephen Atwii Ongom, Jennifer Davis. 2023.
Inter- and intra-rater reliability of handpump functionality field tests. Science of The Total Environment. Volume 869, 161616. https://doi.org/10.1016/j.scitotenv.2023.161616.
Daniel W. Smith, Stephen Atwii Ongom, Jennifer Davis. Does professionalizing maintenance unlock demand for more reliable water supply? Experimental evidence from rural Uganda. World Development. Volume 161, 106094. https://doi.org/10.1016/j.worlddev.2022.106094.
These research projects were funded by the Conrad N. Hilton Foundation.
Rural Water Supply Network - blog 