In advertisement, we often hear slogans like “80% of testers recommend this product”. But do they really? In political trend studies, we hear which parties would be elected into parliament, if people could vote next Sunday. But would the anticipated result really come true? How can we evaluate such numbers and make sense of such empirical data?
Back in 2012 when I started university, I visited a course in economics. In this course, my professor soon explained the most basic understanding about rules and events in economics with a quote that followed me through life ever since. She said: “It depends”.
So, when we look back on numbers that we can perceive in advertisement and our everyday life, we should ask ourselves the same question. Is it true that 80% of testers would recommend a certain product? Or that a party would be elected into parliament with the same share of the votes as attained in a political trend study? Well… it depends.
Although many people might never get in touch with empirical research methods and the reporting of research results, I think anyone can learn some basics that will help making sense of such numbers and data. Therefore, here are the top three insights that, in my experience, will make you take a closer look next time you see such statements.
1. Who has been asked?
If you for example hear that 80% of testers would recommend a certain product, do you automatically believe you would belong to the 80% of the people who like this product? Or may it be the people approving of this product differ from you?
How the number of people or percentage is reported needs to be backed up by additional information. If the question whether or not to recommend a product is positively answered by 4 out of 5, by 8 out of ten, or by eighty out of a hundred, the results show the same percentage.
It is also important to know who has been asked. If the question whether or not to vote for a certain party is only asked to people from one district, the results for the whole country might be very different. The amount of people asked has to be representative for the amount of people the result is ought to apply for. This is called a representative sample.
But when will a sample, the small share of people asked, represent a larger amount of people, the basic population? Couldn’t you simply ask everyone?
To ask everyone would be a total survey or vote itself, which is extremely time and cost consuming. No research or marketing institute would do a survey on such a large scale, which is where statistics comes into action. When you conduct an analysis of your data, statistics can tell you reliable results about your basic population. This rule is bound to some conditions, with one of the most important conditions being the sample is chosen randomly, just to give an example. The randomness enables the possibility that the answers of the people asked are equal to the answers of the basic population.
So, if I question a fair sample, does the result automatically apply to the basic population? Not quite yet.
2. How have they been asked?
If the researcher applied good practice to their sample selection, it is also important to have look at how this sample has been asked. To give a first idea, I will explain two of the possible criteria in empirical research. Both the alteration of scales, and the alteration of survey situations can heavily influence how the survey is conducted.
Thinking back to the product testers’ recommendations, doesn’t it make a huge difference whether the respondents have been asked a yes-or-no-question, or to answer along a spectrum from ‘not likely at all’ to ‘most likely’? It makes a big difference for the data analysis. In the yes-or-no-format it is clear which respondents would recommend a certain product or not. However, when it comes to a more open scale of answers, the researcher already interprets the answers. Following the example, the data would only give an answer about how likely it is that the respondents would recommend the product. Sometimes, the respondents are divided into two groups, between more likely and less likely to recommend the product. It might happen that the two forms, yes-or-no and more-or-less-likely, are confused as one another or the same, when in reality, they are not.
Let’s have another look at the party votes’ simulation. Next to selecting a representative sample, the respondents have to be asked in a comparable situation. A comparable situation means that the respondents’ answers are not dependent on the situation they are in, the moment they are asked. For example, if one researcher asks people on the streets of one big city in one district, and another researcher conducts an online survey, these data sets arose out of different survey situations. It could also occur that one researcher only asks people on one day, while another researcher conducts a survey over a longer time period. This again fabricates different survey situations. The resulting data sets should not be mixed.
So, if I question a fair sample with fair scales in fair situations, does the result apply? Not quite yet.
3. Does this data apply to anyone else?
Last but not least, you can pose the question whether data collected in one country or district will automatically apply to another country or district. Yes, some countries share a large part of their culture and are easily comparable. For example, the comparison between western countries such as the United States and Germany, or between European countries, seem to suggest themselves. Many lifestyle or sports magazines report research results which have been almost exclusively conducted in the US.
However, you might want to observe whether such comparison makes sense in the first place. As we have seen before, representation plays a major role when selecting a sample. And taking it to the next level, a representative sample with representative results do not automatically apply to anyone else, especially if they are not part of the basic population.
So, if I question a fair sample with fair scales in fair situations, does the result apply to anyone else? Maybe.
What’s next after this introduction to empirical research methods
So what’s the next step and how can you apply these learnings to everyday-life? Now that the basics may be clearer, you may remember as soon as you come across a research data result or report to ask some important questions. Who has been asked, how have they been asked, and does the data conducted apply to anyone else?
This is one possible approach from an audience perspective. You can keep in mind that there are many more rules to be applied by researchers themselves. Different study fields also have different research methods. As a percipient, it is important to have a closer look at such reports. Only critical thinking will help you make sense of empirical research. A critical approach will add to your literacy to evaluate scientific research and how it is communicated in media or elsewhere.
Brosius, H. B.; Haas, A.; Koschel, F. (2016). Methoden der empirischen Kommunikationsforschung: eine Einführung (7. Aufl.). Wiesbaden: Springer VS. https://doi.org/10.1007/978-3-531-19996-2
Gerrig, R. J.; Zimbardo, P. G.; Svartal, F.; Brennan, T.; Donaldson, R.; Archer, T. (2012). Psychology and life. CA, United States: Pearson.
Scheufele, B.; Engelmann, I. (2009). Empirische Kommunikationsforschung. Konstanz: UVK Verlagsgesellschaft mbH.