A questionnaire is a commonly used research method to measure the effects of an activity, project, or program. The question is whether a questionnaire is always an appropriate method. Do you want to make statements about the entire target group and collect a lot of data? Then a questionnaire is a smart method. Do you want insight into the underlying motivations and opinions of your target group? Then a questionnaire is a less wise choice. Why is a questionnaire a good method, and why a less good method? The pros and cons explained:
Be creative when choosing your research methods. There are many ways to collect your data. And you can make all sorts of combinations. Think beyond the standard research methods.
By adapting standard methods, you can make it easy for respondents; this will increase your response rate. People will enjoy participating in your research. For example, tear-off cards are very short questionnaires. And short interviews are questionnaires with many open questions. A brief conversation in which a respondent can tell their story over a cup of coffee is a nicer experience than filling out a questionnaire.
Using panels makes good use of people who want to participate in your research, and they are often rewarded for doing so. You pay per respondent, and the panel administrator is happy to help you assess representativeness. You can often select your target group very precisely based on all kinds of background characteristics.
Combining methods gives depth to the information collected. For example, you can interpret the results of a questionnaire in group discussions. Alternatively, you can compile a questionnaire based on a literature review. Or first, analyze existing data files and then ask what is missing in a questionnaire. That way, you don't have to ask several things in your questionnaire, and you can go in-depth about the subject matter in your questionnaire.
You can make respondents keep a logbook and combine this with tracking. This way, you can see what the respondents do, and the respondent also describes it. Please keep the privacy laws in mind.
Combinations I like to use are observations and short conversations based on a questionnaire. Based on what you have observed, you ask some questions. For example, why someone did what they did or how they experienced it.
After formulating your goals, designing the research instrument, and collecting information, it is time to analyze and report the data. Some things to keep in mind when analyzing and reporting your research:
Questionnaires often use scales to measure respondents' opinions but also to look at what they did. However, you can also use scales in observations, logs, tear-off cards, and interviews.
There are several different scales you can use, and they all give slightly different information.
The validity of a study explains the extent to which the questions asked measure what they are supposed to measure. In other words, are the questions asked unambiguous. Could the respondent have interpreted the question differently than you had intended it? And regarding the questionnaire as a whole: do the questions asked in the questionnaire answer the research question.
By challenging a questionnaire, you can find out if it is valid. You can do this by presenting the questionnaire to a test person and having them think out loud while completing it. Another possibility is to present the questionnaire to a test group and have them ask questions as soon as a question is unclear. It is essential that your test persons/group resemble your research population. So if you want to present the questionnaire to young people aged 15 to 18, your test persons/group should consist of young people aged 15 to 18.
After this test, especially with a test group, analyze the answers and ask for advice from a fellow researcher. Are the answers consistent? Is the logic within the questionnaire correct? Are the results consistent with similar studies?
It is more important to test, adapt, and retest with newly developed research instruments. Especially if you want to measure more abstract concepts such as attitude or development, extensive testing is needed to get a valid questionnaire.
When using an existing questionnaire, you should take a moment to see if it has already been tested for comprehension and consistency.
In addition to validity, the concept of reliability is often discussed when conducting research. Want to know more about this? Also read my blog about reliability.
When you conduct interviews, you can find out a lot. You collect primarily qualitative data. (There are some tricks to collect quantitative data as well.) Before you start an interview, it is crucial that you have well-defined interview questions. Because you don't ask questions just for fun: respondents have to answer the central question of your research. That is why I am giving you tips on how to design your interview questions as well as possible:
Finally, I would like to inform you that remaining objective during the interview is essential. Your own opinion plays no role in this; make sure the interviewee can tell their own story.
More tips to prepare for your interview? Then please read my blog '15 tips for a good interview'.
Many people think of interviewing as the widely used method in which the interviewer asks questions to the interviewee. But there are other ways to interview. In this blog, I'll give you some examples.
- #Tip Use a different interviewing method than the widely used question-answer. LINK
- Another new blog is online. Interviewing: What forms are there, and when do you use them? LINK
- Besides the well-known question and answer method, there are other ways you can interview. You can read more examples in this blog: LINK
- With an interview, you don't always have to work with a prepared questionnaire. You can also let respondents get started themselves. LINK
Culture Education with Quality, Network Culture Education and Cultural Participation, Coaching and Training
The reliability of a study points to the extent to which the findings are interpretable to a larger whole. In other words, whether the results represent the entire group and not just the respondents. Did enough respondents complete the questionnaire?
The reliability rate indicates the probability that repeating the survey will produce the same results. With a confidence level of 95%, there is a 95% chance that the answers will be the same with other respondents within this target group. Depending on the type of survey, a reliability rate of 95% to 99% is typically adhered to.
How many respondents are needed depends on how large the research population is. Is the research related to a large group, such as Dutch youngsters between 15 and 18 years old who have participated in a particular project, or a small group, such as public libraries within a certain cao or all inhabitants of Persingen (village with less than 100 inhabitants in Gelderland)? In the case of a large research population, you need more respondents than with a small group. But with a small research population, you need a relatively more prominent part of your research population. I use a practical guideline: with large groups (from 5,000 onward), you need about 400 respondents; for smaller groups, this number increases. The exact calculation depends on various factors, such as margin of error* and the Homogeneity of the research population**. Here I use the sample calculator as a guideline and then stay on the safe side.
Reliability also depends on whether you have asked the right people. Did you only invite people you know to fill out the questionnaire or of whom you know they are satisfied? Or did you randomly select people? In my blog on sampling, I describe how to select the right respondents. Note that not everyone you ask will cooperate in the survey. In addition, a number of questionnaires are always dropped because they are not correctly completed (few questions completed or internally incongruent). Therefore, your sample should be larger than the number of completed questionnaires you need.
* Error margin: The percentage by which the answer may differ from reality. This is similar, for example, to the correction of speed measurements. The speed measurement contains a possible margin of error of 5%, for which traffic control always corrects to the lower value.
** Homogeneity of the study population: The degree to which the members of the study population are similar to each other. For example, students participating in a project are very different from each other (they are about the same age but have very different opinions) and are therefore not a homogeneous research population. In that case, it is better to draw the sample larger. In contrast, groups that voluntarily visit each other and are questioned on a topic will be more homogeneous, and there will be a less extreme divergence of opinions. This, in turn, allows for more detailed questions to be asked.
Suppose you want to research a target group of as many as 10,000 people. Do you have to interview all 10,000 people to get the right results? Certainly not; only a part of the research population needs to participate in your research to get representative results. Let me explain what representativeness means and when the results are deemed characteristic.
Representativeness means the degree to which the respondents in a sample group are a good reflection of the target group of your research. Therefore, your research is representative, which means that the conclusion of your research is true for 'everyone' in your research population.
If you have a research population of 10,000 people, you will ultimately need to interview 400 people to arrive at the opinion of the larger group. This does not mean that you only need to approach 400 people. You have to deal with a response rate. This is the percentage of people who participate in your survey. Your response rate depends on the subject you are researching, how easy and fun it is to participate in your research, and what people get in return. I often use a response rate of 30% because I usually research fun subjects, and I am experienced in making it easy to participate in a survey. I also ask the client for a nice gift for the people who participate in the survey. Tips to increase your response rate
Because not everyone will participate in your survey, you will need to have a larger sample group. If you need 400 respondents and assume a response rate of 30%, you will need to interview a sample group of 400/30%=1,333.
When you conduct a survey, you must keep in mind that the smaller the research population is, the larger the number of respondents will be to arrive at the desired representative results. Sometimes, however, respondents' input is more valuable than the number of participants. In this case, you're referring to qualitative research, and it can be more important to focus on the research results than the representativeness in some cases.
If you want to know how many respondents you need for your research population: go to a sample calculator. These will often immediately tell you how many people you need to approach in your sample.
In a previous blog, I wrote about how best to draw a sample. In the blog, I briefly described the difference between a select and a random sample. There is a substantial difference between the two.
In a random sample, everyone in the focus population has an equal chance of being in the sample. This sample is also called a probability sample or random selection. There are several methods for doing a random sample:
In a select sample, not everyone has a chance to be in the sample group. The results apply only to the group being studied. There are a number of selective sampling options: