In every program evaluation project, constructing the right evaluation question is the most crucial aspect. As I work on a myriad of evaluation projects, where I prepare evaluation plans and provide critical thoughts on other evaluator’s plans, I get the opportunity to develop many evaluation questions. From these experiences, I derived some common themes ‘aka’ strategies to develop evaluation questions. So let me just share some of these strategies that will help you to develop evaluation questions, if you are in the field of monitoring and evaluation-
Stakeholder consultation – The ideal starting point for evaluation questions is to engage stakeholders to hear their perspective. Your stakeholders will act as your ‘brain’s trust’ who will provide rich insight and also play the ‘devil’s advocate’. But it is not wise to talk with a wide number of stakeholders, rather being very selective to nominate the key stakeholders who have bigger influence in the evaluation project. So you have to be very strategic while you line up your stakeholders to discuss the evaluation question. It is a good idea to develop a hierarchy matrix of your stakeholders and nominate them based on hierarchy. But keep in mind that ‘too many cooks can spoil the soup’!
Linking evaluation question with the purpose – It is absolutely pivotal to link your evaluation question with the purpose of the evaluation. You have to narrow down the purpose of the evaluation to a set of key objectives, and from there you have to take out the keywords to refine your evaluation question. It does require some ‘word-smithing’ to develop a final and succinct evaluation question by distilling the purpose and objective of the evaluation. So here is one practical example of an evaluation question from a large state government evaluation project which I was part of.
The government department of a state, wants to improve the emotional intelligence of its employees. As such, it is offering some professional learning to improve the emotional intelligence of the employees.
- Evaluation question –
To what extent professional learning on emotional intelligence is an effective and efficient tool for improving the emotional intelligence of …………… employees?
Attainable measures – The evaluation question should not be too broad, rather, there should be some measures linked with the evaluation question. Whether it is a process, impact or outcome evaluation – you need to think along the lines of measurability. What are those key measures that will determine the success criteria of your evaluation? Your measures will give you pointers of the data that has answers to the evaluation question. For example – if your evaluation question aims to explore financial impact, what measures would be appropriate to qualify as financial impact (eg. financial performance, cost-effectiveness, savings etc.)
Selecting the type of evaluation question – One key considering factor is to determine the type of the evaluation question. If you can not ask the right evaluation question, then you will not get the right data, that will jeopardize the whole evaluation project. In terms of question structure, you might wonder whether an open-ended or close-ended question is right? Open-ended questions are good because they offer you the added room to consider other factors but that can also lead to ambiguity. Open-ended questions are generally better for impact evaluation. But if it is a formative evaluation, then close-ended questions can also be beneficial. The betterevaluation website suggests the following three different types of questions as a rule of thumb –
|Question focus||Considering factors|
|Appropriateness||To what extent does the program address an identified need?How well does the program align with government and agency priorities?|
|Effectiveness||To what extent is the program achieving the intended outcomes, in the short, medium and long term? To what extent is the program producing worthwhile results (outputs, outcomes) and/or meeting each of its objectives?|
|Efficiency||Do the outcomes of the program represent value for money? To what extent is the relationship between inputs and outputs timely, cost-effective and to expected standards?|
Linking evaluation question with the evaluation type – There are various types of evaluation and each evaluation type has a specific pattern. For example -the process evaluation is all about how things are working now, the appropriateness of the program etc. Whereas, the impact evaluation is all about the outcome, overall effect etc. The economic evaluation will seek to answer the financial effectiveness, best use of resources and so forth. Therefore, the evaluation type will entirely determine the evaluation question. I find the following table from betterevaluation website very useful to guide the evaluation question corresponding to the evaluation type –
|Type||Typical key evaluation questions|
|Process evaluation||How is the program being implemented? How appropriate are the processes compared with quality standards? Is the program being implemented correctly? Are participants being reached as intended? How satisfied are program clients? For which clients? What has been done in an innovative way?|
|Outcome evaluation (or impact evaluation)||How well did the program work? Did the program produce or contribute to the intended outcomes in the short, medium and long term? For whom, in what ways and in what circumstances? What unintended outcomes (positive and negative) were produced? To what extent can changes be attributed to the program? What were the particular features of the program and context that made a difference? What was the influence of other factors?|
|Economic evaluation (cost-effectiveness analysis and cost-benefit analysis)||What has been the ratio of costs to benefits? What is the most cost-effective option? Has the intervention been cost-effective (compared to alternatives)? Is the program the best use of resources?|