In terms of methodologies, there are few simpler than a survey. They ask people what they think directly and clearly. They tend to trust in people’s answers and rely on people’s honesty. Historically this has been known to work. At the same time, that doesn’t mean making a survey is completely effortless. There are a lot of differences between a good quality survey and a bad one.
One of the biggest considerations in survey creation is the questions. This seems obvious on some level. Surveys that aren’t asking the right questions are clearly going to be flawed. Yet that doesn’t just mean asking relevant questions, it means asking them in the right ways. For example, it’s important to avoid loaded questions to have good data. The difference between “How often do you drink these brands of soda” and “Do you drink soda” is massive.
The first makes an assumption while the second doesn’t and also sets up for a later question. If they drink soda, the next question is “Which of these soda brands do you drink on a weekly basis?”. Asking questions properly takes more time to come up with and answer. It can seem like a small difference but data is useless if not obtained correctly. Loaded questions are just one example of a common mistake, but the takeaway is to be conscious.
It’s also helpful, even before making questions, to have a plan. This means knowing who to survey, where, when, and how. Is the survey being conducted online, is it going to take up a lot of time? These are essential questions. They can be put off until later, but this can really delay the actual deployment of the survey massively.
The most important question and one that must be answered in advance is the why. Each survey should have an explicit purpose which leads to its questions and analysis. Without this there can easily be irrelevant populations surveyed and questions asked. Luckily if that’s all done, the survey can be conducted and the data obtained.
Of course, it doesn’t end there. Data analysis and reporting may be the single most challenging step. It’s here that even the best data can become entirely useless or ruined. This is why some companies opt to outsource this work. There are vendors that explicitly work to code and analyze data. It can be as simple as how much money one wants to spend and how much experience they have.
For teams or companies with no experience in statistics, it’ll be hard to learn quickly and effectively. There’s a steep learning curve, especially with no external support. At the same time exporting can be expensive and requires a lot of revisions. Fully exporting really moves deadlines out of the survey creator’s hands. There may be expected deadlines but it’s ultimately not in the creators control.
This is just part of what makes the analysis phase of survey creation so challenging and demanding. Even once the data is properly analyzed it must still be formatted and displayed. This offers a whole host of new issues when most people who look at the data will only see the graphs. It’s here again that each team has to consider who will be doing this and how.
Luckily formatting is much more reasonable to pick up and learn. Although the better the graph the more likely it will have an impact. There are good reasons to value something as simple as formatting really highly. Big errors such as using the wrong type of graph can completely cripple the effect of a survey. The harder something is to understand, the less likely someone is to interface with it at all.
Beginning to end, there are complications with survey creation. None of this is meant to scare survey creators, they are just things to keep in mind. In reality many surveys will be incredibly short, simple, and easy to make. It’s for those that are looking to have robust, articulate, and impactful surveys that these complications most apply. Researchers in particular are, of course, considering these questions constantly. Yet for those looking to make a more casual survey, remember that good planning and questions will carry any project.
