Crowdsourcing: Fresh thinking or online fad?

Results are encouraging, but long-term effectiveness remains uncertain. 

Companies that don’t know about crowdsourcing risk falling behind, especially if their success relies on new products or innovative ideas.

The concept of getting non-experts to solve problems or do small tasks for free or for the chance to win a prize has been around since the 1700s, when the British government offered the Longitude Prize to whoever could come up with a reliable way for sailors to navigate when their ships got too far from land. Today the Internet makes crowdsourcing much cheaper and easier to implement and extends its reach to diverse populations. While computers can perform some tasks exponentially faster than cerebral synapses, businesses still rely on the human brain for nuanced thinking and creativity.

Marketing professor Barry Bayus is exploring the theory and application of crowdsourcing to learn its benefits and limitations.

“Crowdsourcing won’t replace R&D departments anytime soon,” Bayus said. “While it can give valuable information in certain situations, the research I’m doing shows there are some issues to be aware of.”

Bayus analyzes Dell Computer’s IdeaStorm community in “Crowdsourcing New Product Ideas Over Time: An Analysis of the Dell IdeaStorm Community,” a forthcoming article in Management Science. Originally conceived to draw fresh ideas from customers and the general public (in response to the “Dell Hell” customer service controversy), IdeaStorm costs the company little while strengthening its connections to customers and potential customers.

Dell launched its IdeaStorm website in 2007 as a sort of online suggestion box where anyone can share their ideas for new products, software or services or other means of improving the consumer’s experience. Occasionally, an idea piques Dell’s interest to the point that it implements the idea. Contributors post suggestions, knowing that if Dell should commercialize them, the contributor would receive no royalties nor claim to the intellectual property. No remuneration other than, say, a nice pen-and-pencil set. Still, contributors posted a storm of ideas and became an ongoing online community, voting for their favorites and commenting on the submissions of others, and submitting new suggestions as inspiration hit.

Despite the excitement that companies show for crowdsourcing, little formal research has been done to examine whether the quantity and quality of ideas can be sustained over time, who contributes the most and best ideas, and what effect incentive structures have on creativity. Bayus took on the challenge by scraping the data publicly available on Dell’s website since its inception. (These data involve only ideation activities of individuals – no personal information is collected.) He has collected and analyzed two years’ of data on thousands of ideas and individuals using advanced statistical methods. This approach allows him to control for other possible explanations involving unobserved differences in creativity or skill among community members. His results highlight challenges in maintaining an ongoing supply of good ideas from the crowd over time.

The launch of IdeaStorm met with thousands of suggestions in the first few months, followed by a steep decline in activity and then leveled off to modest participation. The most interesting behaviors were at the individual level, Bayus found.

• The vast majority of people propose a single idea, and Dell found few that it wanted to implement.
• Most good ideas came from people who proposed more than one idea — serial “ideators.”

“One of the key challenges for companies with crowdsourcing communities is to get members to fully engage and participate, rather than sit on the fence and watch,” he said. They need to recruit new participants and encourage them to submit many ideas. In Dell’s case, Bayus found evidence that, between 2007 and 2009, the well of new ideas from these ideators might be drying up.

“You don’t want to ignore the people who aren’t giving you good ideas right now,” Bayus said. “Keeping active participation over time is hard, because typically a lot of people may join, but very few people actively participate.”

Bayus looked closely at the behavior and suggestions of people who submitted many ideas. Did they produce better ideas over time? Were more of their ideas implemented the longer they participated in the IdeaStorm community? Was past success positively related to the chance of proposing yet another good idea that excited Dell? Surprisingly, that was not the case.

Good ideas don’t continue to come from people who already contributed a winner, he found. “My research found that once someone comes up with what Dell considers a good idea, that person tends to fixate on the implemented idea,” Bayus said. “Their subsequent ideas aren’t as creative.” He surmised that people narrowed their thinking to propose similar ideas they believed might interest Dell, rather than submitting unconventional ideas that might lead to bigger improvements.

Bayus suggests that companies track the number of individuals proposing an idea and compare it to the number of people proposing an idea for the first time. Similar values for these two metrics indicate that new members are not becoming serial ideators – and even serial ideators tend to become fixated on their successes and propose good ideas only over a relatively short period of time.

One way to break this fixation tendency, Bayus found, is to encourage commenting on a diverse set of ideas proposed by others. Research suggests that simply instructing participants to not focus on prior submissions might be sufficient to stimulate their creativity.

Bayus has started to study whether incentives could be used to generate creative and innovative ideas. He is collaborating with one of the largest international co-creation communities that sponsors online idea contests for major clients such as Coca-Cola, Gillette, Carrefour, Danone, Philips and Honda to gather empirical research that add new lines of thought to the theoretical models of contests and incentives studied by economists.

Mathematical models posit that offering one big prize (as opposed to several small prizes) generates the greatest level of effort and the best outcome because it will attract the most skilled contestants and discourage the least skilled. If the contestant pool is heterogeneous, including professionals and amateurs, offering several small prizes attracts more effort from less-skilled people, while the more-skilled people aim for the large prize.

Early results from Bayus’ empirical research add further insight. He found that large prizes led to more contest entries, yet prizes were not related to the creativity of submissions.

“I really don’t think you can pay people to be more creative,” Bayus said. “Truly creative people are intrinsically motivated by the fun or feedback they get. Extrinsic motivators like money are usually a hindrance to creativity.”More interestingly, he found that winner-take-all contests with a single large prize generally end up with relatively fewer creative ideas, despite more entries.

“It seems that the chance at one large prize gets a lot of people excited, including pros and especially amateurs,” he said. “Unfortunately, the large number of submissions from amateurs only results in a lot of uncreative ideas.”

Bayus also is very interested in the mechanisms for filtering ideas in crowdsourcing systems.

“Right now, most companies still are manually evaluating the hundreds of ideas that might come from a crowdsourcing initiative,” he said. “You certainly don’t want to overlook a great idea, but can the crowd help out?”

Very little research has taken up this question. For Bayus, the right data and company involvement can make further progress in rigorously finding an answer.

After all his research, Bayus still is not completely convinced that crowdsourcing merits the excitement it has generated. However, he believes that studying real-world applications will help make a case one way or another.

“Very few crowdsourcing systems have lasted more than a couple of years,” he said. “They have no established history of successes or failures, so little is known about their long-term effectiveness.”