7.7: Common categories of evaluation questions

Most questions raised about programs are questions about: needs, process, outcomes or impact. Turn each card to learn more about that area of evaluation.

This graphic shows how these questions fit with the logic model:

Five additional elements have been added the basic logic model to illustrate how evaluation questions fit into the logic model process. Underneath the logic model, a yellow box labeled "Process" extends across the entire logic model; superimposed on that box are three triangles. On the left, a small triangle labeled "Needs" points to the Situation component of the logic model. On the right, a large triangle labeled “Outcomes” extends across the space corresponding to the Outcomes-Impact component. This triangle points to the Outcomes-Impact component. On the far right, a narrow triangle labeled “Impact” points to the Long Term section of the Outcomes-Impact component. In addition, a purple oval labeled "Satisfaction" is linked to the Outputs (Participation) component of the model.

What about participant/customer/client satisfaction?

In the logic model, satisfaction is connected to the outputs, in the middle of the model. This differentiates it from many models, such as Total Quality Management (TQM), where customer or client satisfaction is the apex of performance.

In theories of change, client satisfaction may be necessary, but it is not sufficient for outcomes to occur. For example, a participant may be satisfied with the program and express positive reactions such as “I liked the program,” “It fit my needs,” “I will come again.” But such satisfaction does not mean that the person learned anything or can do anything differently, or that life has improved for the person as a result of the program. Satisfaction may indicate that a person is likely to fully participate in and complete a program. The learning environment can be an important factor contributing to changes in knowledge, attitudes, motivation, etc. Satisfaction, however, does not measure the results achieved.

Formative and Summative Evaluation Questions

As you learned earlier in this section, a program can be evaluated at any time. Questions that can be asked in a program’s life cycle fall into formative and summative categories. Formative and summative are common words in evaluation.

Formative Questions

Formative questions are asked during the program–while the program is operating. They may be asked on an ongoing basis or at periodic times over the course of the program’s life. The questions are usually asked for the purpose of program improvement–to receive immediate feedback and input in order to know how things are going and what improvements–corrections and/or additions–might be needed.

To what extent are the parents that we targeted for this program attending? Are they completing the program?
Are all youth participating in all sessions? If not, why not?
Are the mentors spending the expected amount of time with the students?
Do people appear to be learning?
What seems to be working, not working? For whom? Why?

Summative Questions

Summative questions ask about what resulted, what was effective. These questions are asked at or after completion of the program (or a phase of the program). They are asked largely for the purpose of deciding whether to continue, extend, or terminate a program.

To what extent did communication problems decline as a result of the cross-cultural training program?
Do participants shop differently as a result of their participation in the program? How?
Given the results, was the program worth the costs?

Formative and summative are not synonymous with process and outcome. Formative and process occur during the program’s early stages and focus on improving the program; summative and outcome focus on what happens to participants/community/environment at the conclusion of the program or program phase. However, formative and summative relate to intentions–to collect data for ongoing program improvement or for decisions about program continuation or termination. Process and outcomes refer to the phase of the program being studied. You might ask formative or summative questions at any phase of a program’s development cycle.