Thursday, January 26, 2017

Recommender system failure as a business model: Repellent ad? Pay for premium!



While writing my last post, I spent a lot of time worrying about whether we really understand the forces at play with so much of our information world driven by business models based on clicks. My underlying assumption was that these forces all perpetuate the dependence of the production and flow of information in today's world on advertising. Today, I was reminded of the importance of thinking out of the box, and never assuming anything: there might be exceptions.

Here's what's happening: YouTube incentivized me to subscribe to YouTube Red by showing me an ad that raised the hair on the back of my neck, and then giving me a pop up window asking "Want to remove ads?" (screenshot above).

Specifically what happened: the pre-roll ad for my video was from Urban Carry Holsters:


G2 Overview from David Foster on Vimeo.

and while my video played I had Urban Carry Holsters videos suggested at the upper right hand of my page:

After watching this for a while in horrified fascination, YouTube opened a pop-up:


The "try it" button might as well have been labeled: "Get me out of here!"

Pretty brilliant, really. What I am assuming is happening  (i.e., "may be could be happening") is that the recommender system algorithm is optimized to increase not only the number of ad clicks, but also the number of YouTube Red subscriptions.

Of course, I am a proponent of recommender systems that are not designed to fulfill a single target [1]. The target could be ill-designed, and the world is also just not that simple.

However, I am of two minds about what YouTube just did to me as a user. First, when we talk about gun violence in the US, we talk about deaths and causalities. The discussion of the psychological wear and tear is often in the shadows. If my heartbeat rises with an ad like this one, then I can't even imagine what parents must go through, who send their kids out the door in the morning to school with the constant worry of stray bullets and guns in irresponsible hands. Ads like these just contribute to the second-order harm that the fact that we have no real gun solution inflicts on society. YouTube's recommender should know enough about me to protect me from the psychological wear and tear (which results in wasted time).

Second, maybe YouTube should not be protecting me, but exposing me to more. (Yes, I am of two minds, and the second is completely opposite.) If recommender systems recommend advertisements that are personalized to be repellent for users, it could be a force that drives subscriptions at a large scale. If enough other people react like me, we will soon be on the road to being able to fund the production and distribution of information based on quality and trust, funded by subscriptions, rather than on clicks.

There is a chance that this ad is not a complete recommender system misfire. The Urban Carry Holster ad was not actually an utter mismatch for my tastes. They show that the holster was designed on the basis of a "user study", and I have certainly purchased a number of high quality real leather handbags in my day. It's the "detail" of putting the gun inside of it that freaked me out.

So maybe it is a recommender system failure, or maybe it is the most important thing that recommenders have done for our online information ecosystem in years. Whichever of these points of view ends of winning, it is something worthwhile thinking about.

My only concern is the manipulation aspect: in order not to destroy trust with YouTube, I would appreciate knowing that the ads are optimized to increase YouTube Red subscriptions, and I am indeed being nudged.

[1] A Said, D Tikk, K Stumpf, Y Shi, M Larson, P Cremonesi. 2012. Recommender Systems Evaluation: A 3D Benchmark. ACM RecSys 2012 Workshop on Recommendation Utility Evaluation: Beyond RMSE, Dublin, Ireland

Sunday, January 8, 2017

Down the Rabbit Hole: Greetings from a state of extreme information overload

This morning, I innocently checked the news. Then I disappeared down the rabbit hole. One click following another, driven by the idea that around the next bend I would arrive at some kind of a lasting understanding that would outlive today.

When I realized I was in full information reading free fall, I started writing this blog post, just to record what was happening.

To reconstruct the beginning of the experience, I asked myself what was the lead story on The Guardian when I opened it this morning.

Do I really remember what happened two hours ago? First, I thought no. Then, I remembered it was something about the shooting in Fort Lauderdale. But what? The shooter was unhappy in some way. Let me go back to check, but whoops in the meantime, there is a fully different lead story...I can't go back to where I was...maybe Fort Lauderdale was not so important after all.

Actually, no don't want to be reading about Fort Lauderdale. I land in Florida airports rather frequently and I don't need to be creating anxiety. Shouldn't be reading that one.

Spent some time trying to get back to see the same "first page" that I saw this morning...clock ticking. It appeared not to be possible.

What kind of insight will I arrive at that will outlive today?

This morning became this afternoon as I dove into certain column with the headline: "Moral panic over fake news hides the real enemy – the digital giants"

https://www.theguardian.com/commentisfree/2017/jan/08/blaming-fake-news-not-the-answer-democracy-crisis

Hmmm. What exactly is "moral panic"?

I read this:

https://www.psychologytoday.com/blog/wicked-deeds/201507/moral-panic-who-benefits-public-fear

Interesting. We learn:

"Moral panic has been defined as a situation in which public fears and state interventions greatly exceed the objective threat posed to society by a particular individual or group who is/are claimed to be responsible for creating the threat in the first place."

However, that gets us nowhere on what "Moral panic over fake news hides the real enemy – the digital giants" is going to actually tell us. If there is a fear, it is related to the fact that we have no way of estimating an objective threat, and by this definition can't be moral panic.

OK. Title doesn't make sense. Let's click anyway. Maybe this article will allow me to move forward on one of my more dominant streams of thoughts these days: The discourse on news and news reading behavior seems to assume that people have an unlimited amount of time and attention resources to consume news in a given day. How do we achieve a healthy and balanced news diet, if we don't have countless hours to spend?

This stream of thought has led me to ask the question if the time that we are spent worrying about "fake news" should be spent thinking about something else. And the related question: "What is that something?" and "Is the problem with fake news actually not that it is fake but that it is simply consuming time that we should be spending doing other things?"

So I click. Falling, falling. The piece is interesting, but not what I expected.

Yet I am reading ideas that I don't recall encountering before in such a form. I keep reading. Second to last paragraph is:

"The only solution to the problem of fake news that neither misdiagnoses the problem nor overpowers the elites is to completely rethink the fundamentals of digital capitalism. We need to make online advertising – and its destructive click-and-share drive – less central to how we live, work and communicate. At the same time, we need to delegate more decision-making power to citizens – rather than the easily corruptible experts and venal corporations."

But how much does the author really know about the forces at play within the larger context that gives rise to online advertising? If there is going to be a "rethinking" there need to be "rethinkers" who are positioned to make changes. This piece seems to be implying that those "rethinkers" exist: but can they exert the required influence?

OK. I could fall forever. I am just going to dig a bit more deeply into this one article, and then I am going to stop and do something else.

Let's start with remembering exactly does "venal" mean again? Looked that up. "Open to bribery". Right. OK.

To understand who the author might consider to be the "rethinkers", let's have a look at where the author is coming from, specifically, what he might know about neuroscience and psychology, i.e., information addiction and confirmation bias, information literacy, and the science of complex systems. I started out by looking at the profile page of the author, here:

https://www.theguardian.com/profile/evgeny-morozov

which links to his blog here:

http://neteffect.foreignpolicy.com

Which doesn't give me a blog, but rather a portal:


what is going on?

I decide to read all of the comments to see if anyone else had this problem.

Whew. Lot's of opinions there. Pretty interesting discussion. One comment states "We have norm of unexamined adoption". That's an interesting observation: How did those norms get formed in the first place? If we can figure that out, then we can maybe take some action there.

At least two people commenting are pointing to the need for helping people develop critical thinking thinking skills and the ability to verify information. That's another of my streams of thought lately: how to promote the practice of evaluating information sources, for example, with the CAARP test.

No one seems to be bothered by the broken link in the author profile of the author of the piece. Usually a broken link would point to a poorly maintained, and potentially less authoritative source. But this is The Guardian! Maybe I am seeing things?

Then I spend some time on The Guardian website trying to figure out where to report a broken link. Lots of opportunities for suggesting corrections to content, and for securely passing The Guardian information. Good to see. However, none for just saying, "Hey, the link is bad".

OK. Time to take action. I posted the comment:

"Does anyone else find that the link to Morozov's Net Effect blog at the top of his Guardian profile page (https://www.theguardian.com/profile/evgeny-morozov) doesn't seem to really lead to the blog? It seems like the The Guardian made a mistake, and that the link should be directing us here: https://foreignpolicy.com/category/net-effect Uncertainty about this link is hindering me in digging into the wider context of this piece."

Time passes.

Worrying that that comment will be interpreted as being negative about the piece. I'm not negative, just trying to get to the bottom of what the lasting message is for me.

Time passes.

I'm spending time on trying to understand why research on dopamine and information seeking seems to have fallen silent in the mainstream press after 2012, and on wondering why there is not a good website to explain complex systems. We need to rely on Wikipedia for so many of the related concepts like "preferential attachment" and "emergence".

Why in the world does my Morozov piece feature one picture of Putin and the top and one picture of Trump in the middle? It is not about either of them. I don't think Morozov chose those.

I am still falling...with also a feeling of having been sucked in.

Time passes.

This is about one piece that I read in the newspaper! I'm trying to form an opinion about one single opinion piece. What if I had tried to read the other ones as well? What if I were doing any serious fact checking?

Greetings from a state of extreme information overload.

Time passes.

Is the conclusion that the limits our time will ultimately always win? That we will drown in a state of information overload because it requires an afternoon to evaluate a single opinion piece?

I am not so sure. In this case, I am planning to take action on my conclusions regarding the article and the things that the people are saying in the comments. As an information retrieval researcher, a crisis in information quality is a crisis at the core of our research field. As an instructor of a freshman information science course I need to be able to describe best practices in information and consumption behavior.

There is a lot riding on this one article for me.

In that respect, it is not wasted time.

A arrive at the bottom of with a loud bump.

So the conclusion is, yes, our time is limited. We can't spend an entire afternoon examining everything that we read. The most important information is the information that we take action on. We need to seek out that information, and evaluate the heck out of it.

If we are not planning to take action, read, but leave the information in suspended animation. For example, the article on Fort Lauderdale. Or: there is now an article about the Mob on the front page of the New York Times. I choose not to subject these to scrutiny, but neither will I take any action (including sharing those articles) on the basis of what I read.

Looking at the length of this post, another obvious conclusion is that people should set aside more time for finding and consuming information. The information available online initially looks "free", but really we need to also count the price of our time. Information without verification is useless.

Setting aside time requires asking the question, "What did we lose because we didn't choose to do something else instead?"In short, how can we more tightly link reading the news to tradeoffs and to tangible value?

Now what about this little bottle?

Alice drink me

Friday, December 16, 2016

Advanced Bullshit Detection for your protection: Wise words on reading the news

There's been lots of news on the news in the news recently. Just like we try to take care of our bodies by eating healthy foods, we must take care of our societies by carefully consuming quality news. The importance of a high-quality, balanced media diet is independent of your political convictions. 

But how do we keep our news reading habits healthy? What do we do? This morning I came across a great video explaining four steps that everyone can take in order to achieve a healthier news diet. The video is an interview with Curd Knüpfer at the Freie Universität Berlin published by Die Zeit with this article. Since the video is in German, I provide my own translation of the four steps here. I tried to make the translation as accessible as possible.  

I call it "Advanced Bullshit Detection": these four steps are what we need to be spending our time doing in order to protect ourselves while reading the news.

As a media consumer, you have to develop an attitude towards yourself so that you see yourself as someone who chooses news cautiously. There are guidelines that you can follow, and that you can make into habits:

1. Ask yourself questions about your own emotional reaction
When I see an article or a news items that that makes me particularly angry, one to which I have a very strong emotional reaction, one that makes me nervous or fearful, then I should stop and think. I should stop and ask "Wait a minute, what is this information actually saying, and why is it having this effect on me?"

2. Check the quality standards
Develop an eye for what good, meaningful journalism is, and how news articles are created. This means that one can look at any form of journalistic reporting (it doesn't matter where it is from or its political direction). You can say "It's better if an article on a particular topic cites more than one source" or if more than one source is cited, "It is better that more than one perspective is represented."

3. Pay attention to the sources of the information
Of course you need to pay attention to the sources. Can I, for example, trust someone who works for the Freie Universität Berlin? And, if I think that I can't: Why can't I? It also works the other way around. You can say, "Hey, there's someone who knows this topic relatively well!" It doesn't mean that you take everything they say at face value, but at least you can trace back where the person is from and who is paying them, etc.

4. Balance your media diet
Balancing your media diet is a luxury that we have because we live in a world in which media is digital. It is relatively easy for us to access a large number of different sources of news, and we should also take advantage of the diversity available to us.

Thank you Curd Knüpfer  for these wise words (did my best not to lose anything in translation).

Sunday, November 13, 2016

Big Data as Fast Data

Last Thursday I was at a "sounding board" meeting of the Big Data Commission of the Royal Netherlands Academy of Arts and Sciences. This post highlights some points that I have continued to reflect upon since the meeting. 

According to Wikipedia, "Big Data" are data sets that are too large and too complex for traditional data processing systems to work with them. Interestingly, the people who characterize "Big Data" in terms of volume, variety and velocity, often underemphasize, as the Wikipedia definition does, the aspect of  velocity. Here, I argue it is important not to forget that Big Data is also Fast Data.

Fast Streams and Big Challenges
Because I work in the area of recommender systems, I quite naturally conceptualize problems in terms of a data stream rather than a data set. The task a stream-based recommender system addresses is the following: there is a stream of incoming events and the goal is to make predictions on the future of the stream. There are two issues that differentiate stream-based views of data from set-based views.

First: the temporal ordering in the stream means that ordinary cross-validation cannot be applied. A form of A/B testing must be used in order to evaluate the quality of predictions. Online A/B testing has implications for the replicability of experiments.

Second: at any given moment, you are making two intertwining predictions. One is the prediction of the future of the stream. The other is how much, if any, of the past is actually relevant in predicting the future. There are two reasons why the information in the past stream may not be relevant to the future: external and internal factors.

External factors are challenging because you may not know they are happening. A colleague doing medical research recently told me that when deductibles go up people delay going to the doctor, and suddenly the patients that are visiting the doctor have different problems, simply because they delayed their visit. Confounding variables of course exist for set-based data. However, if you are testing stream-based prediction online, you can't simply turn back the clock and start investigating confounding variables: it's already water under the bridge. As much as you may be recording, you cannot reply all of reality as it happened.

Internal factors are even tougher. Events occurring in the data stream influence the stream itself. A common example is the process by which a video goes viral on the Web. In this case, we have a stream of events consisting of viewers watching the video. Because people like to watch videos that are popular (or are simply curious about what everyone else is watching) events in the past actually serve to create the future, yielding an exponentially growing number of views. These factors can be understood as feedback loops. Another important issue, which occurs in recommender systems, is that the future of the stream is influenced by the predictions that you make. In a recommender system, these predictions are shown to users in the form of recommended items, and the users create new events by interacting with these items. The medical researcher is stuck with this effect: she cannot decide not to cure patients, just because it will create a sudden shift in the statistical properties of her data stream.

Time to Learn to do it Do it Right
In short, you are trying to predict and also to predict whether you can predict. We still call it "Big Data", but clearly we are at a place where the assumption that data greed pays off ("There's no data like more data") breaks down. Instead, we start to consider the price of  Big Data failure ("The bigger they are, the harder they fall").

In a recent article Trump's Win Isn't the Death of Data---It Was Flawed All Along, Wired concluded that "...the data used to predict the outcome of one of the most important events in recent history was flawed." But if you think about it: of all the preposterous statements made during the campaign, no one proposed that the actual election be cancelled since Big Data could predict its outcome. There are purposes that Big Data can fulfill, and purposes for which it is not appropriate.

The Law of Large Numbers forms the basis for reliably repeatable predictions. For this reason, it is clear that Big Data is not dead. The situation is perhaps exactly the opposite: Big Data has just been born. We have reasons to believe in its enormous usefulness, but ultimately its usefulness will depend on the availability of people with the background to support it.

There is a division between people with a classic statistics and machine learning background who know how to predict (who may even have the technical expertise to do it at large scale) and people who, on top of a classical background, have the skills to approach the question of when does it even make sense to be predicting. Only the latter are qualified to pursue big data.

The difference is perhaps a bit like the difference between snorkeling and scuba diving. Both are forms of underwater exploration, and many people don't necessarily realize that there is a difference. However, if you can snorkel, you are still a long way from being able to scuba dive. For scuba diving, you need additional training, and more equipment, and a firm grasp of principles that are not necessarily intuitive, such as the physiological effects of depth and the wisdom of redundancy. There is a lot to be achieved on a scuba dive, that can't be accomplished by mere snorkeling: but the diver needs resources to invest, and above all needs to have the time to learn to do it right.

No Fast Track to Big Data
These considerations lead to the realization that although Big Data streams may be in and of themselves incredibly quickly changing, the overall process of making Big Data useful is, in fact, very slow. Working in Big Data requires an enormous amount of training going beyond a traditional data processing background.

Gaining the expertise needed for Big Data also requires understanding of domains that lie outside of traditional math and computer science fields. All work in Big Data areas must start from a solid  ethical and legal foundation. Civil engineers are in some cases able to lift a building to add a foundation.  With Big Data, this possibility is excluded.

To illustrate this point, it is worth returning to consider the idea of replacing the election with a group of data scientists carrying out Big Data calculations. It is perhaps an extreme example, but it is one that makes clear that ethical and legal considerations must come before Big Data. The election must remain the election because on its own a Big Data calculation has no way of achieving the necessary social trust necessary to ensure continuity of the government. For this we need a cohesive society and we need the law. Unless Big Data starts from ethics and from legal considerations, we risk time and effort developing a large number of algorithms that are solving the wrong problems.

Training data scientists while ignoring the ethical and legal implications of Big Data is a short cut that is tempting in the short run, but can do nothing but harm us in the long run.

Big Data as Slow Science
The amount of time and effort needed to make Big Data work, might lead us to expect that Big Data should yield some sort of Big Bang, a real scientific revolution. In fact, however, it are the principles the same old scientific method of centuries that we return to in order to define Big Data experiments. In short, Big Data is a natural development of existing practices. Some have even argued that data-driven science pre-dated the digital age, e.g., this essay entitled Is the Fourth Paradigm Really New?

However, it would also be wrong to characterize Big Data as business as usual. A more apt characterization is as follows: Before the Big Data age scientific research proceeded along a the conventional path: researchers would formulate their hypothesis, design their experiment, and then as the final step collect the data. Now, the path starts with the data, which inspires and informs the hypothesis. The experimental design must compensate for the fact that the data was "found" rather than collected.

Given this state of affairs, it is easy to slip into the impression that Big Data is "fast" in the sense that it speeds up the process of scientific discovery. After all, the data collection process, which in the past could take years, can be carried out quickly. If the workflow is implemented, a new hypothesis could be investigated in a matter of hours. However, it important to consider how the speed of the experiment itself influences the way in which we formulate hypotheses. Because there is little cost to running an experiment, there is little incentive to put a great deal of careful thought and consideration into which hypotheses we are testing.

A good hypotheses is one that is motivated by a sense of scientific curiosity, and/or societal need and that has been informed by ample amounts of real-world experience. If there is negligible additional cost to running an additional experiment, we need to find our motivation for formulating good hypotheses elsewhere. The price of thoughtlessly investigating hypotheses merely because they can be formulated given a specific data collection is high. Quick and dirty experiments lead to mistaking spurious correlations for effects, and yield insights that fall short of generalizing to meaning real-world phenomena, let alone use cases.

In sum, we should remember the "V" of velocity. Big Data is not just data sets, its also data streams, which makes Big Data also Fast Data. Taking a look at data streams makes it easier to see the ways in which Big Data can go wrong, and why it requires special training, tools, and techniques.

Volume, variety, and velocity have been extended by some to include other "Vs" such as Veracity and Value. Here, I would like to propose "Vigilance". For Big Data to be successful we need to slow down: train people with a broad range of expertise, connect people to work in multi-skilled teams, and give them the time and the resources needed in order to do Big Data right. In the end, the value of Big Data is the new insights that it reveals, and not the speed at which it reveals them.

Tuesday, October 18, 2016

The Societal Impact of Multimedia Research: ACM MM 2016 Brave New Ideas Track (Papers and Slides)


At ACM Multimedia 2016, the Brave New Ideas Track was devoted to the theme "Societal Impact of Multimedia Research". In the Call for Papers, we challenged authors to be brave in pursuing topics that have a direct impact on people's lives. 

What is brave about multimedia research with societal impact? The answer is simple: it takes much more time. To pursue work with direct societal impact it is necessary to work together with other disciplines, create new data resources, and develop new evaluation methodologies that demonstrate success with respect to socially relevant criteria. 

It also takes time for researchers to reach the insight that important new scientific questions arise from directly attempting to create solutions for societal problems. For example, concerns about privacy are currently motivating researchers to turn away from pursuit of "data greedy" algorithms to study how to get more out of less data.

We recommend reading the papers of the track to understand other interesting scientific problems that have been opened up by researchers who have the courage to work in these high societal-impact areas:

Mengfan Tang, Siripen Pongpaichet, and Ramesh Jain. 2016. Research Challenges in Developing Multimedia Systems for Managing Emergency Situations. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 938-947. [ACM DL link][slides]

Andrea Castelletti, Roman Fedorov, Piero Fraternali, and Matteo Giuliani. 2016. Multimedia on the Mountaintop: Using Public Snow Images to Improve Water Systems Operation. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 948-957. [ACM DL link][paper][slides]

Alexis Joly, Hervé Goëau, Julien Champ, Samuel Dufour-Kowalski, Henning Müller, and Pierre Bonnet. 2016. Crowdsourcing Biodiversity Monitoring: How Sharing your Photo Stream can Sustain our Planet. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 958-967. [ACM DL link][paper][slides] See also: the Pl@ntNet App.

Michael Riegler, Mathias Lux, Carsten Gridwodz, Concetto Spampinato, Thomas de Lange, Sigrun L. Eskeland, Konstantin Pogorelov, Wallapak Tavanapong, Peter T. Schmidt, Cathal Gurrin, Dag Johansen, Håvard Johansen, and Pål Halvorsen. 2016. Multimedia and Medicine: Teammates for Better Disease Detection and Survival. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 968-977. [ACM DL link][paper][slides]


Contact: ACM MM 2016 BNI Chairs: Martha Larson (TU Delft and Radboud University Nijmegen) and Hari Sundaram (University of Illinois)

Tuesday, August 16, 2016

Music as Technology: Sad song of missed opportunities for music to do what music does well

Last week my colleagues Andrew Demetriou and Cynthia Liem presented a paper at ISMIR 2016 (the 17th International Society for Music Information Retrieval Conference) entitled "Go with the Flow: When Listeners Use Music as Technology" [1]. The idea of this paper is that listeners use music as a tool that is directed to accomplishing a task.

In the paper, we point to the phenomenon of people using music to put themselves into a flow state: "listeners make a conscious decision to expose themselves to the experience of music to alter their internal state in order to achieve a goal that they have set for themselves." With this paper, we want to encourage the development of music information retrieval technology, including recommender systems, that supports listeners in finding the music that they need in order to support their goals.

It is a chicken and egg problem. Unless systems are there that support users in finding music that allows them to reach there goals, it is hard to study the phenomenon at large scale. Unless we understand the phenomenon, it is hard to develop these systems. The first leap remains one of, well, basically, faith: faith that, given the evidence that we already have on hand, that we should push for music information retrieval that recognizes that positive potential of music for allowing people to best use their brains.

We try to keep the focus on the positive potential, but the dark underside of the situation is what happens if our world continues on, with the mainstream being unaware of the effect of music on the brain. The wrong music can prevent certain brain states, as easily as the right music can promote them. When music is not in the control of an individual (such as in a restaurant, cafe, or public place) serious thought is needed about what music is playing and how to play it. Otherwise the music is putting people in a brain state that is neither productive, nor even pleasant.

Stop the noise by Surko
In Chicago, I met a man cleaning tables in a restaurant.

The chain had obviously put a lot of time and money into the decor.

The music was a loud mix of alternating genres. Understanding people speaking was a strain.

When I asked the man about it he got emotional.

"I'm teaching myself guitar," he said. "There are a couple of good country pieces mixed in, but I know them by heart. The whole thing plays over and over again."

"I could tell them what they really should be playing!"

We look to a future in which the people making the music decisions about places like this restaurant care about their sound atmosphere as much as they care about the visual impression of their decor. The music should not only be geared to customers who stop by for lunch, but also to employees, who spend their days listening to it. The experience (and arguably also mental health) of people spending time in the restaurant could be enhanced if the music released them from the grind of repetition. Music may not always allow people to achieve flow state, but it can support them in enjoying being where they are, as much as possible.

Music decision makers should sit up and realize that music matters: music doesn't happen "out there" somewhere, but rather happens inside every person who is listening to it. The leap of faith is not a large one, it just requires listening to people who know that music is important, and asking them how to make things better.

[1] Demetriou, A., Larson, M., and Liem, C. Go with the Flow: When Listeners Use Music as Technology, ISMIR 2016.
https://wp.nyu.edu/ismir2016/wp-content/uploads/sites/2294/2016/07/068_Paper.pdf

Thursday, July 28, 2016

How to use the word "subjective" in multimedia content analysis

Multimedia content analysis is devoted to the automatic processing of video, image, audio, and text content with the purpose of describing it, or otherwise associating it with information that will make it findable, and also useful, to users. Previously, I have urged multimedia content analysis researchers to avoid the word “subjective” and instead formulate their insights in terms of inter-annotator agreement with respect to the data that they are using and the protocol that they give to the annotators who are providing the target labels. Since we don’t seem to be inclined to stop using the word “subjective” soon, it makes sense to formulate some guidelines on how to use it "safely".

Best practice for the use of the word “subjective”: When the word "subjective" it is used, it should be first defined.

The word "subjective" has different definitions. It’s not particularly productive to fix any one way of using it as “the only right way”. Instead, when using the word "subjective" you should simply declare which definition you are using, and you will avoid a lot of unproductive confusion. You do not want to risk that you use “subjective” in one sense, and your reader/listener interprets it in another sense.

,

We can gain further understanding of why it is important to "define well before use" by examining the dictionary entry for “subjective” provided by Merriam-Webster. Here, you can see the many meanings that “subjective” can take on. I haven’t observed any issues caused by definitions 1 or 2. Multimedia content analysis research is generally not interested in these definitions. Where we get into trouble is with 3-5, so I will focus on these.

Let’s start with definition 4c: “arising out of or identified by means of one's perception of one's own states and processes” This definition of subjective is related to the conceptualization of a situation as being exclusively determined by the point of view of the “subject”, i.e., the person who is undergoing the experience of perceiving something.

Such a conceptualization, in the case of certain situations, is standard, and when we communicate with each other, we don’t even think about the fact that we assume it.  Let’s take a closer look at how this conceptualization works. When we use language, we rely on an unspoken agreement that certain phenomena (for example, the emotion that music evokes in a person) are subjective. Specifically, the agreement means that the way in which we understand the world gives all listeners the power to determine what they feel when listening to music (i.e., induced emotion) for themselves.

Simply stated: if someone says, “This music makes me so happy”, it is nonsensical for me to assert, “No, it doesn’t”. I might say this to tease someone, but it is clear that I am not using language in a standard way. An emotion felt while listening to music can only be asserted by the subject, and I, who am not in the subject’s mind, do not have the power to originate a meaningful statement on the matter. It is not a trivial point: Without this shared understanding, the convention/assumption of subjectivity behind "This music makes me happy", the function of language would break down and we would have failed to communicate.

Here’s where things can go wrong for a researcher working in the area of multimedia content analysis. Imagine you are collecting multimedia content labels from a group of annotators who are judging the content, and you at the end of experiment, and declare, “The results show that the phenomenon we are studying is subjective”. Readers who are using definition 4c of subjective will find this conclusion invalid. The reason is that under this definition, “subjective” is something that is established ahead of time by convention: it cannot be determined experimentally. (Full disclosure: for me this is the preferred definition of "subjective", because it is the most literal interpretation. The word "subjective" contains the word "subject". I also prefer it since it ensures the sanctity of the private world of the individual, and the right of the individual to an independent voice.)

Moving to 4b: “arising from conditions within the brain or sense organs and not directly caused by external stimuli” This definition is not so interesting for multimedia researchers: we study multimedia content, which is an external stimuli. 

Now, we go on to definition 4a: (1): “peculiar to a particular individual:  personal” This definition of subjective is related to the idea that each individual has their own unique view. (Merrian-Webster's Definition 1 of "peculiar" is "characteristic of only one person, group, or thing") Under this definition, something is "subjective" it means that everyone disagrees with everyone else. This definition is also not so interesting for multimedia researchers: if everyone has their own completely different interpretation, then we are lost: we cannot hope to build algorithms that generalize over the different meanings that find in multimedia. Until the field of multimedia starts working extensively on systems used only by a single person, this definition of subjective is probably not one that will be used often.

Note that the field of recommender systems strives to develop personalized algorithms, and users evaluation methodologies that assess whether personal predictions are successful. However, even recommender systems rely on the fact that people are similar to each other. In a world populated exclusively with utterly unique individuals, collaborative filtering algorithms will necessarily fail.

More helpful is definition 4a (2): “modified or affected by personal views, experience, or background” This definition of "subjective" is often implicitly assumed in multimedia content analysis. People’s interpretations are affected by what they know, the opinions they hold, and the life experience that they have had. These factors can lead to there being a multitude of different interpretations that apply to certain multimedia content. However, in contrast to the situation above with definition 4a (1), we are not assuming that everyone has their own “peculiar” interpretations. It makes sense for us to try to create systems that generalize or predict meaning, only in the case that we are not dealing with exclusively unique interpretations.

We can see 4a (2) as closely related to 3b: “relating to or being experience or knowledge as conditioned by personal mental characteristics or states”

With both of these definitions, 4a (2) and 3b, we can reasonably have hope that we can find islands of consistency in the perceptions of users of multimedia (and in the labels of our annotators). Within these islands we can make stable inferences that will be useful to users.

Let’s check again if, under these definitions, you can make a statement in your paper, “The results show that the phenomenon we are studying is subjective”. This time you can. But in order to do so, you need to have an experiment that shows that the background of the users is what is causing your classifier not to give you stable predictions. Otherwise, it might be the case that your classifier just has not been well designed or trained.

You also need to provide evidence that the protocol that your annotators are using to make judgements is not unduly steering people to diverse interpretations. Your protocol should put people reasonably on the same page, and then ask them for judgements at all times being careful not to ask "leading" questions, cf. [1, 2]. For some research work, you might not be using a protocol. Many tasks involve "found" labels such as tags. In this case, you need to state the assumptions that you are making concerning the original labeling context, including the reasons for which the labels were assigned.

With any definition of subjective, it is important to strictly avoid arguing along these lines: “This phenomenon is subjective, and therefore it is not important and we should not be studying it.” 

Scientifically, there is no a priori reason to prioritize the “objective” over the “subjective” if we use definitions 4a (2) and 3b.  It is true that we tend to study phenomena with high inner-annotator agreement since these are easier to get a handle on. However, at the same time we remain aware that this tendency steers us dangerously close to the famous story of Nasreddin Hodja who looks for his ring outside, since it is too dark inside where he lost it. In short, define “subjective”, but never use it as an excuse for failure or avoidance.

To drive that particular point home: The message is "Keep up your guard". Your problem should arise from the needs of users. Practically, speaking the problem you choose will be influenced by your ability to access the resources needed to study it, including carrying out a well designed, conclusive experiment. It will not, however, be influenced by your personal decision that something is "subjective".
  
Next, we turn to definition 3a: “characteristic of or belonging to reality as perceived rather than as independent of mind.” Using this definition is dangerous. It forces you to take a position on the difference between effects that are real, and effects that are imagined. As scientists, we determine this difference experimentally. We do not presume it. Unless we are undertaking experiments directed a making this difference, it makes sense to steer clear of this definition.

Finally, definition 5: “lacking in reality or substance” The same comment applies as in the case of definition 3a. We cannot a priori say whether patterns that can be found in multimedia content lack reality or substance.  If we don’t find evidence for the reality of some phenomenon in our data, it simply means that there is no evidence for its reality in our data. Lack of observation does not disprove existence. We must guard ourselves against jumping to conclusions. Again, this is a definition to be avoided, unless you are actually directly investigating the nature of reality.

As researchers in the area of multimedia content analysis, we must carefully keep ourselves from creating our own realities: the reality we assume must be the reality (possible multiple realities) of the users that we serve—all of them. The fact that we do not necessarily understand this reality fully, or have the type of information or data that would capture it in its complexity, richness, and continuous, rapid evolution, is a challenge that we face. This challenge is inherent to the types of algorithms and technologies that we design and develop.


[1] Larson, M., Melenhorst, M., Menéndez, M. and Peng Xu. Using Crowdsourcing to Capture Complexity in Human Interpretations of Multimedia Content. In: Ionescu, B. et al. Fusion in Computer Vision – Understanding Complex Visual Content, Springer, pp. 229-269, 2014.

[2] M. Riegler, V. R. Gaddam, M. Larson, R. Eg, P. Halvorsen and C. Griwodz, "Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks," 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI), Bucharest, 2016, pp. 1-6.