After working for almost twenty years in development projects, I now realise project implementers have very often misunderstood the success of their projects. Unfortunately, in a donor-grant implementer or client-subcontractor relationship, everyone seems to be more interested to see if their plans are successful in achieving the intended objectives, hoping that the funding would continue in the future. Donors and funders are not that different. The perceived power they have over grant implementers, subcontractors or development partners is artificial. They do have politicians to report to, or for charity organisations; they have the people who donate their money to share good stories. This situation makes those involved in development projects susceptible to positive bias, whereby you are tempted to jump into a conclusion that your efforts have paid off, and your projects are successful, thus have brought about changes to the people as initially intended.
A small-holder rice farmer in Bojonegoro, East Java – Indonesia
Biased by the intention to expect for good news, we tend to be less critical in determining if the projects have indeed caused the changes observed among the participants. We are too quick to celebrate the success and dub them as the prominent achievements of the project. Trying to establish if a project causes the observed change deals with the research methods in proving the causal relationship between the project and the outcome. In my field of work, this relates to the Monitoring and Evaluation field. An area that coins two different concepts of measurement but then obscures the real meaning of each term. To me, the Monitoring and Evaluation field is as simple as, borrowing a vast body of knowledge already available in, research and experimentation, and apply it in social and development programs. I know it would not be as rigour as it would be in research, but we could do is to approximate it. In research, trying to understand if a project causes an observed change is about determining a causal relationship between one variable over the other. This requires us to understand factors that may affect our conclusion of a causal relationship, or commonly known as threats to internal validity. I will elaborate on real examples of how these threats can be found in various development projects.
- Ambiguous Temporal Precedence – it happens when there is a confusion of which variable occurs first and therefore makes it unclear about which one is the cause and the effect.
- Selection – this bias occurs when there are systematic differences over conditions in respondent characteristics that could also cause the observed effect. This happens when, at the start of the project, the participants could be already different than the average target beneficiaries. A project that targets to increase primary school children’s knowledge of science may consist of children whose parents devote more time in helping their children to learn and create supportive learning atmosphere at home with all the materials. These children will be likely to score better than those who do not have the same characteristics even though they are exposed to the same project activities.
- History – This bias exists when other events occur between the beginning of the project implementation and the time when we observe changes among the beneficiaries. Such events could also explain the outcomes even when there is no project implementation. A project that aims to increase rural women’s financial literacy through training could perceive the project as a success. Little that the project members realise, banks and other microfinance institutions are also aggressive in undertaking various outreach education to these women. The outcomes are then doubtful if it is really the project that helps increase their understanding of saving and lending activities or instead of the interaction with these other players.
- Maturation – Sometimes, changes observed among the participants are just natural as they become older or more experienced, and project implementers may confuse these changes as a result of the activities. A psychosocial support project for women victims of domestic violence may confuse the improved emotional conditions of these women are due to the support provided. Yet, with the absence of the project, these women may also recover on their own naturally.
- Regression – When participants of our project are chosen because of the extreme scores, there is a possibility that they may be able to score better in the next test but not necessarily due to the project intervention. Supposedly, the same children who participate in the science class, in the baseline, some of the children could have extremely low scores. The next test reveals that the same children could score better, but these achievements are not necessarily due to our interventions. There is a tendency, if the first score is too extreme, the second measurement may reveal a higher score just by chance even when there is no treatment implemented. As program implementers, we need to be careful about this situation when there is a tendency of the scores to regress around the mean.
- Attrition – this bias occurs when some of the participants (thus the subject of our measurement) drop out of the project, and those who remain do not necessarily reflect the real condition of the whole group. A nutrition project that intends to help mothers to be aware of good feeding practice may find that some of these mothers drop out before the project completes. Those remaining in the project actually have had a different habit of feeding their children compared to the ones dropping out. Observation on the women who stay in the project may satisfy the project implementers and think that their project has indeed improved the mothers’ behaviour, but in fact, mothers who need the intervention the most have already dropped out of the project.
- Testing – Taking a test repeatedly may actually influence our target participants to get better, not necessarily because our intervention works. In my line of work, exposing the scholarship recipients to numerous opportunities to take language exams may actually help the participants improve their score just by merely gaining experience from taking these tests. We could misunderstand the improved scores as improved language capacity due to the training we conduct.
- Instrumentation – Tools that we use to measure may change over time, and we could misinterpret the changes due to program intervention. A nutrition project that aims to increase the weight of children under two years old could find that the weighing scale used is not as accurate as it was during the baseline because of wear and tear condition. The spring becomes loose because of over-use when weighing hundreds of children. Enumerators who conduct an estimation of farmers’ yield during the baseline get better and become more experienced during the end line survey. Yet, the difference does not reflect the situation that the farmers increase their harvest, it is just because the enumerator are getting smarter with the measurement.
- Additive and interactive effects – this occurs when more than one factor is affecting the bias interpretation of the causal relationship. Very often, the above threats do not operate in isolation. You could find a project is affected by selection and history bias at the same time. Farmers who get involved in an agriculture training project may come from those who are already competent in farming and during the program timeframe they also participate in other activities outside the project that help them more knowledgable about the use of fertilisers. These two situations shroud the real effect of the training program that we conduct.
I guess this writing becomes a reflection of what I have done or seen in the past that we are too quick to be complacent of the results on the ground. Putting our utmost efforts and investing our time in the activities makes us fall in love with the projects, and therefore, we want to believe that our projects are in the end beneficial to our target beneficiaries.
This is natural, but what if such a situation persists and gets repeated over time, how many ineffective projects are out there misunderstood as successful? What if the same unproven project concept receives new funding repeating the same kind of ineffective treatment? Are we actually doing more harm than good?
So next time before you give yourself a pat on your back, please be critical of those factors above, reflect if any of those threats exist in your project.
More in-depth readings:
Shadish, W.R. Cook, T. D. Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston Newyork. Houghton Mifflin Company
A lighter version of the same concept is also explained in;
Weiss, C. (1997). Evaluation. Upper Saddle River, NJ, United States. Pearson Education.