Music as Technology: Sad song of missed opportunities for music to do what music does well

Last week my colleagues Andrew Demetriou and Cynthia Liem presented a paper at ISMIR 2016 (the 17th International Society for Music Information Retrieval Conference) entitled "Go with the Flow: When Listeners Use Music as Technology" [1]. The idea of this paper is that listeners use music as a tool that is directed to accomplishing a task.

In the paper, we point to the phenomenon of people using music to put themselves into a flow state: "listeners make a conscious decision to expose themselves to the experience of music to alter their internal state in order to achieve a goal that they have set for themselves." With this paper, we want to encourage the development of music information retrieval technology, including recommender systems, that supports listeners in finding the music that they need in order to support their goals.

It is a chicken and egg problem. Unless systems are there that support users in finding music that allows them to reach there goals, it is hard to study the phenomenon at large scale. Unless we understand the phenomenon, it is hard to develop these systems. The first leap remains one of, well, basically, faith: faith that, given the evidence that we already have on hand, that we should push for music information retrieval that recognizes that positive potential of music for allowing people to best use their brains.

We try to keep the focus on the positive potential, but the dark underside of the situation is what happens if our world continues on, with the mainstream being unaware of the effect of music on the brain. The wrong music can prevent certain brain states, as easily as the right music can promote them. When music is not in the control of an individual (such as in a restaurant, cafe, or public place) serious thought is needed about what music is playing and how to play it. Otherwise the music is putting people in a brain state that is neither productive, nor even pleasant.

Stop the noise by Surko

In Chicago, I met a man cleaning tables in a restaurant.

The chain had obviously put a lot of time and money into the decor.

The music was a loud mix of alternating genres. Understanding people speaking was a strain.

When I asked the man about it he got emotional.

"I'm teaching myself guitar," he said. "There are a couple of good country pieces mixed in, but I know them by heart. The whole thing plays over and over again."

"I could tell them what they really should be playing!"

We look to a future in which the people making the music decisions about places like this restaurant care about their sound atmosphere as much as they care about the visual impression of their decor. The music should not only be geared to customers who stop by for lunch, but also to employees, who spend their days listening to it. The experience (and arguably also mental health) of people spending time in the restaurant could be enhanced if the music released them from the grind of repetition. Music may not always allow people to achieve flow state, but it can support them in enjoying being where they are, as much as possible.

Music decision makers should sit up and realize that music matters: music doesn't happen "out there" somewhere, but rather happens inside every person who is listening to it. The leap of faith is not a large one, it just requires listening to people who know that music is important, and asking them how to make things better.

[1] Demetriou, A., Larson, M., and Liem, C. Go with the Flow: When Listeners Use Music as Technology, ISMIR 2016.
https://wp.nyu.edu/ismir2016/wp-content/uploads/sites/2294/2016/07/068_Paper.pdf

Multimedia Bits

Cynthia Liem recently gave a TedX Delft talk called "Every bit of it". I have the pleasure and the honor of being her colleague at the Delft Multimedia Information Retrieval Lab. I missed being at the talk in person, but finally found time today to watch it on YouTube.

What she is saying is so critical---it at the same time both fresh and timeless---that the talk deserves a more in-depth reaction than a tweet or re-tweet. This post summarizes the message that I heard in Cynthia's talk.

Cynthia makes the point that every bit on the Internet has meaning to the original person who put it there. These are of course, multimedia bits, that she is speaking of videos and images that have been captured by people and shared on the Internet. An important question in this era of Big Data: each of these "bits" of multimedia is captured by someone for some reason. Cynthia tells us that we can add worth to these bits by enhancing them with other bits; in particular, she shows us wondrous transformations that can be brought about by adding music to video.



Towards the end of the video, she says that a big question that with face with multimedia is "What is relevant?" http://youtu.be/ykkXdjr0rfI?t=14m1s

She points out that we tend to focus for the obvious in our understanding of relevance, e.g., popularity and quality, and that because of this focus, we miss material that people do not know exists.

Instead, the "bits" of multimedia on the Internet should be seen as an entrance to the world, not as the final product, but a place to start. A diamond in the rough that needs to be polished in order to add value.

At the end of the video, she asks the audience to keep their eyes open for a rediscovered hidden gem....I discuss my gem in my next post: http://ngrams.blogspot.nl/2013/03/multimedia-readymades.html

Peaceable Kingdom: Snowflake button for YouTube snow

A post that mixes the magic and delight of the holiday season with multimedia information retrieval? Let's try it and see what happens.

The past couple of weeks holiday cards have been dropping through the mail slot in the front door -- but also emails have been entering my inbox: greetings, photos, and yes, also videos. This morning it was an email with a greeting and a link to a music video "Peaceable Kingdom".

I watched the video for a while and pondered its relationship with Christmas: The music is melodious, soothing and the lyrics take the listener to the manger to make the connection with the adoring state of mind of those who gathered there the first Christmas Eve. Unexpected minor cadences highlight that this is no usual Christmas carol and invite consideration of the multiplicity of the Christmas experience -- how the holiday itself integrates traditions preceding Christianity and how, as each new group and generation reinvents it for their own spirit and needs, it will continue to develop into some future Christmas. From the perspective of the here and now, that future Christmas could seem full of sweetness, hope and light, but also distorted and distinctly pagan.

Of course, the strongest signal I get from the video is that of Margaret Atwood's dystopic visions. I haven't read The Year of the Flood, but what has been written and said about the book has so much fascinated and disturbed me, that the existence of the book as itself as a text seems somehow less important to me -- the setting is already so palpable that what it tells is, in a way, no longer left to be said.

In the end, maybe my personal Christmas feeling associated with the video is that it gives me a chance to spend some time feeling close to the person who sent it to me. The strength of this feeling of connection goes beyond -- indeed exists in a completely different life dimension -- than my reflections on meta-text usurping text or on the length of time that has transpired since I have sat down and read a worthwhile book not related to work.

Where is the multimedia information retrieval tie-in? Well, first, as a result of this video it has occurred to me for the umpteenth time that we need a verb other than "watch" to describe this kind of interaction with this video. It's a music video, so I am mainly listening to it and then looking at the visual stimuli. There could potentially be rather large changes in the visuals -- different pictures, different editing -- and these changes could possibly leave my watching experience largely untouched. I would argue, if I were only "watching", these elements would necessarily have a major defining impact on my experience. They don't. Here, I am rather "watch/listening", which I suppose could give us the new concept of "wistening".

There's a second tie-in as well: There is a little snowflake in the player bar, which I discovered after "wistening" for a while. I usually find snowflake icons ambiguous: especially on climate control units in strange hotel rooms -- do I turn the setting to "snowflake" if it's cold outside or is the "snowflake" setting going to cause the system to start producing cool? I've encountered both. So I've learned just to click on the snowflake and see what happens...

I clicked.

And lo and behold it started snowing. Right into the Peaceable Kingdom -- flakes floating down slowly -- different sorts of flakes at different speeds -- and accumulating at the bottom of the frame. I felt the smile spread on my face -- and grow wider as a realized that I was witnessing one little bit of a sort of world-wide holiday miracle as people in front of screens around the planet discover that you make it snow on YouTube. I thought about people watching this on their laptops and tables, using the mouse to play a bit in the snow and then gathering their friends, colleagues, family around their screens in one big Christmas "You gotta check this out!"

Apparently, you can't do this to every video: and this is where it really starts getting interesting to me. How did YouTube decide which videos to add this feature to? There must have been some multimedia classification algorithm that maybe looked for keywords in the title and description and something like music in the audio channel or colors in the visual channel and combined this with the upload date -- and then enabled "snow" for this video.

I want to make these kinds of algorithms! How do we put everything that we know how to do in terms of multimodal video processing and machine learning and figure out for which videos it needs to be able to snow?

And it's not just snow. There are other ways in which this could go -- and should go -- it has potential to cause so much joy. I am sitting here "wistening" and thinking about friends and family and playing in the snow, but it's clear that we need to go being "wistening" and we need a very for watching+listening+reflecting+playing. It's also clear that we need the technologies that support these activities. Imagine a search engine that can find videos that are appropriate for 'snow': that goes so far beyond user information needs as they are currently conceptualized for multimedia that it sort of takes your breath away.

How to enable the multimedia community to work at these new (from the perspective of this moment, utterly fantastic) frontiers?

The key to doing work in this direction, is to evaluating it. How do we know if we were right in presenting the snow option for a given video? YouTube is probably analyzing its interaction logs at this very moment. But I hate to think that I need to go to work for YouTube in order to ever be able to do the evaluation necessary to write a paper on this topic. Everyone loves the snow, so everyone should be able to work in order to make it better.

Note to self qua New Year's resolution: Keep up commitment to evaluation -- we need it to push ourselves forward into the unknown in a meaningful way. Maybe it's what actually makes the difference between what we call computer science and what we call art. But I'll leave that thought to another day.

In the meantime, the overall conclusion is that holidays and multimedia information retrieval do indeed mix well in a blog post. So happy holidays (ans enjoy the video):