CrowdRec 2015 Workshop Panel Discussion: Crowdsourcing in Recommender Systems

The CrowdRec 2015 Workshop on Crowdsourcing and Human Computation for Recommender Systems was held this past Saturday at ACM RecSys 2015 in Vienna, Austria. The workshop ended with a fish bowl panel on the topic of the biggest challenge facing the successful use of crowdsourcing for recommender systems. I asked the panelist to take a position as to the nature of this challenge, was it related to algorithms, engineering or ethics. During the panel the audience took crowdsourced notes about the panel on titanpad.

After the workshop I received two comments that particularly stuck in my mind. One was that I should have told people that if they contributed to the titanpad notes, I would write a blogpost summarizing them. I was happy that at least someone thought that a blog post would be a good idea. (I hadn't considered that having ones thoughts summarized in my blog would be a motivational factor.) The other comment was that the panel question was not controversial enough to spark a good discussion.

In response to these two comments here now a summary/interpretation of what was recorded in the crowdsourced notes about the panel.

• The biggest challenge of crowdsourcing is to design a product in which crowdsourcing adds value for the user. Crowdsourcing should not be pursued unless it makes a clear contribution.

• The way that the crowd uses a crowdsourcing platform, or a system that integrates crowdsourcing is essential. Here, engagement of the crowd is key, so that they are "in tune" with the goals of the platform, and make a productive contribution.

• The biggest challenge is the principle of KYC. Here, instead of Know Your Client, this is Know Your Crowd. There are many individual and cultural differences between crowdmembers that need to be taken into account.

• The problem facing many systems is not the large amount of data, but that they data is unpredictably structured and in homogenous, making it difficult to ask the crowd to actually do something with it.

• With human contributors in the crowd, people become afraid of collusion attacks that go against the original, or presumed. intent of a platform. A huge space for discussion (which was not pursued during the panel) opens about who has the right to decide what the "right" and "wrong" way to use a platform.

• Crowdwork can be considered people paying with their time: We need to carefully think about what they receive in return.

• With the exception of this last comment, it seemed that most people on the panel found it difficult to say something meaningful about ethics in the short time that was available for the discussion.

In general, we noticed that there are still multiple definitions of crowdsourcing at play in the community. In the introduction to the workshop, I pointed out that we are most interested in definitions of crowdsourcing where crowdwork occurs in response to a task that was explicitly formulated. In other words, collecting data that was create for another purpose rather than in response to a taskasker is not crowdsourcing in the sense of CrowdRec. It's not uninteresting to consider recommender systems that leverage user comments collected from the web. However, we feel that such systems fall under the heading of "social" rather than "crowd", and reserve a special space for "crowd" recommender systems, which involve active elicitation of information. It seems that it is difficult to be productively controversial, if we need to delineate the topic at the start of every conversation.

At this point, it seems that we are seeing more recommender systems that involve taggers and curators. Amazon Mechanical Turk, of course, came into being as an in-house system to improve product recommendations, cf. Wikipedia. However, it seems that recommender systems that actively leverage the input of the crowd still need to come into their own.

See also:

Martha Larson, Domonkos Tikk, and Roberto Turrin. 2015. Overview of ACM RecSys CrowdRec 2015 Workshop: Crowdsourcing and Human Computation for Recommender Systems. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). ACM, New York, NY, USA, 341-342.

Towards Responsible and Sustainable Crowdsourcing

Humans are the ultimate intelligent systems. Units of human work can be used to address the problems studied in the fields of pattern recognition and artificial intelligence. After years of research to crack certain tough problems, mere utterance of the phrase "human cycle" makes it seem like someone turned on a light in the room. Suddenly, we feel we are no longer feeling our way forward in darkness as we develop solutions. Instead, a bright world of new possibilities has been opened.

The excitement that crowdsourcing has generated in computer science is related to the fact that large crowdsourcing platforms make it possible to apply abstraction to human input to the system. It is not necessary to consider who exactly provides the input, or how they "compute" it, rather the human processor can be treated as a black box. The magic comes when it is possible to make a "call the the crowd" and be sure that there will be a crowdworker there to return a value in response to that call.

However, crowdsourcing raises a whole new array of issues. At the same time that we excitedly pursue the potential of "Artifical artificial intelligence" (as it's called by MTurk), it is necessary to also remember "Human human computation".

I am not an ethicist, and my first foray into crowdsourcing ethics was relatively recent and necessarily superficial. In fact, I started by typing the word "ethics" into my favorite mainstream search engine and picking a definition to study that seemed to me to be authoritative. However, I am convinced that the community of crowdworkers and taskaskers together form an ecosystem and that the main threat to this ecosystem is that we treat it irresponsibly.

In other words, we should not throw out everything that we have learned over centuries of human civilization about creating healthy and happy societies, stable economies and safe and fulfilled individuals in our quest to create new systems. Ultimately, these systems must serve humanity as a whole, and not disproportionately or detrimentally lean on the portion of the population that serves as crowdworkers.

Because of this conviction, I have put together a set of slides about responsible crowdsourcing that serve as notes on the ethical aspects of crowdsourcing. At a recent Dagstuhl seminar entitled, "Crowdsourcing: From Theory to Practice and Long-Term Perspectives" I used the slides to make a presentation intended to serve as a basis for opening a discussion on the ethical issues of crowdsourcing.

The hopeful part of this undertaking is that it revealed many solutions to address ethical aspects of crowdsourcing. Some of them pose challenges that are just as exciting as the ones that motivated us to turn to crowdsourcing in the first place.

Please see the references in the slides, and also these links:

http://crowdwork-ethics.wtf.tw
http://www.slideshare.net/mattlease/lease-statisticsethics