Podcast: Karine Pepin on finding value in survey incompletes

We recently welcomed Karine Pepin, Vice President at 2CV, to discuss the concept of incomplete or partial data. She sets the stage by talking about how rapidly the world of insights is changing due to things like shorter attention spans overall and the massive impact of technology, with ResTech market research solutions transforming the industry across the board from data collection, to survey platforms and data analysis tools. We shouldn’t underestimate the profound impact this has had on so many aspects of our work, including how we should be treating incomplete data in our dataset.

She says that there is a “traffic jam” in the survey response collection process. The demand for surveys has increased while the willingness to take surveys has decreased. In addition, there have been more fraud and quality issues which means that many of our completes are tossed out in the back-end. There is a shortage of good participants, and we continue to experience persistent problems with sample quality.

Still, survey conversion is still the ultimate metric. We cannot afford good panelists who have qualified to not finish. Every panelist counts – literally and figuratively. So we have to optimize for the new reality to not only encourage respondent engagement, but also consider the value in the incomplete responses. To boost engagement at the front end, Karine says our first line of defense is to provide a better survey experience so people stay. This can include practices like shorter surveys that are device agnostic, which should be matter of course. We need to make questionnaires more engaging overall.

She says that in an ideal world, we want to create an experience where the participant looks forward to seeing the next question, like a book. There is a lot of conversation about how to make the “questions” better or to gamify questions but she encourages thinking about “storification.” This concept is more about improving the structure and the flow of the survey at a macro-level. Some elements of that include providing an enticing landing page, improving the user interface, using illustrations, icons, emojis, using a more conversational tone, and offering a sense of progression. 

But despite our best efforts, people will still inevitably drop out. This is where the idea of keeping partial completes comes in. Karine mentioned “Every participant counts and it kills me to see people drop-out especially when they are almost done. I always ask myself ‘what could have I done better’?”  Depending on the sample size,  there will usually be hundreds of participants who dropped out at various points in the study. 

When you keep partial completes, you can actually help reduce the traffic jam, speed up fieldwork and value the participant’s voice even if they didn’t finish. And the quality of the data is often high, because the respondents went through the same entry quality checks as those who completed. While the demographics of people who dropout are not always the same, they are typically people who are less engaged with the category and could provide valuable nuances in the findings.  For example, in a consumer electronics study, it could be older females dropping out. On a gaming study for a casual game designed for women, it could be male players dropping out. This means that we certainly always skew the data to more frequent / involved / higher spender consumers by only keeping completes. 

She says that, of course, she still prefers a complete dataset. And the value of keeping partial completes varies from study to study. On a high incidence study, it would be best to simply wait until you hit your quotas with completed interviews. If the incidence is low and replacing these individuals is difficult, then it might be a study where keeping partials makes sense. Keeping partial completes can create inconsistency in the data, so it is something to be aware of when examining the overall results.  

Karine acknowledges that even 10 years ago, we would not have considered keeping any incomplete data to feed into our research results or insights. We didn’t need to because we could get plenty of respondents, and they would be more likely to finish a long survey. Today, she says we are facing new challenges. We have to be creative in how we approach these challenges and evolve with the participants and the technology.