September 27, 2016

Why small samples matter in Conversation Research

 

Conversation research is distinct from traditional market research in that it is largely unstructured. We use a variety of softwares and tools to process the data sets to produce some degree of organization – topics, sources, themes, etc. – but you’re not pulling a sample of 1000 people of a certain demographic and asking them the same questions here. You’re casting a wide net looking for similarities in random conversations from around the world.

So when your review comes back with 100 conversations out of 23,000, it’s easy to dismiss this percentage (less than 0.02) as not valid. But let’s look at an example and see if validity needs to be reconsidered.

CRI recently conducted a high-level scan of the online conversations around work-life balance with our friends at Workfront. The project management software company focuses a lot of its content on work-life balance as its solution helps bring that result to marketing agencies and brand teams around the world.

Over the 30-day period ending September 19, we found 23,021 total conversations on blogs, social networks, news sites, forums and more – essentially any publicly available online source where people could post or comment – about work-life balance.

If you focus on the 23,021 as your total pool of conversations, it might frustrate you that only eight percent (1,827) could be automatically scored for sentiment. (One can manually score much more, and CRI typically does a fair amount of work to close that gap, but it is an exercise in time and resources that for this project both parties elected to set aside.)

But if you take that eight percent – those 1,827 conversations – and now consider them your sample set, you’ve got something. There, we discover that 79 percent of the scored conversations were positive – people are generally in favor of or have good reactions to the concept of work-life balance. But that means 21 perent of them don’t.

And this is where our curiosity is piqued.

 

It turns out the predominant driver around the negative conversations on work-life balance is that the concept itself is a myth. Out of the 382 total negatively scored conversations found, 98 of them indicated in some way that work-life balance was a lie, a farce, an illusion and so on.

Another 10 were tied to a conversation around a piece of content exposing the “lies” of work-life balance, also indicating there’s some level of mistrust that it is attainable. And 10 more revolved around a reference to work-life balance being overrated.

So while the negatively scored conversations were just 0.02% of the total conversation set, they were 21% of the total subset that could be scored for sentiment. And of that subset, more than one quarter were focused on the concept not being real at all.

This is where deeper analysis can help us synthesize true insight. Why do people think it’s a myth? Is it that the naysayers are likely cynics who cannot draw hard lines between their work time and focus and that which they spend away from work? Or do the demands of most jobs actually make it impossible to separate work from life? Or is it something else?

The bottom line is that one shouldn’t be dismissive of small data sets from big data, especially when it comes to conversation research. Remember that while we may only be talking about 100 conversations out of 23,000, but those 100 conversations are from people who are proactively discussing the topic at hand, not people being led down a Q&A path by an interviewer or a survey.

This brings delightful structure to that unstructured world.

0 Comments

Leave A Comment

Leave a Reply

Interested in learning more?
Subscribe to our free newsletter and get a free style advice every week. We will also notify You about new offers and discounts. Check it!