Event Detection in Multimedia: Different definitions, different research challenges.

Yesterday, someone asked me for a pointer to work in the area of event detection in multimedia content. This mail prompted me to finally get out a blog post that explains the distinction between the different sorts of underlying challenges that researchers are referring to when they discuss events in multimedia.

A simple definition of an event is a triple (t, p, a), consisting of a moment a time t, a place in space p, and one or more actors a. For example,  at (t=) 2pm 13 July 2013. at the (p=) Faculty of Electrical Engineering, Mathematics and Computer Science at Delft University of Technology, (a=) I am now involved in a "blog-post writing" event.

Let's look at that definition of event in terms of aspects that matter to us as multimedia researchers. If you took a video of me (here and now), another human looking at your video might notice that I am also eating a salad. Consequently, the video could equally be considered to depict a "lunch-eating event". For this reason, it makes sense to also introduce a fourth variable "v", to arrive at (t, p, a, v). The "v"stands for the name for the action or the activity part of the event. I use "v" for "verb" since these names corresponds to verbs or can be expressed by phrases involving verbs.

Note that there are at least three basic ways to name (or label) events when it comes to multimedia: (1) name the event from the perspective of the/an actor (I, the actor, call it a lunch eating event, because I know it is lunch.) (2) name the event from the perspective of the person recording the multimedia (The person sees me engaged in an eating event, but do not necessarily know or care that it is lunch.) and (3) name the event from the perspective of the/a person looking at the multimedia in a time and place other than when and where it was captured (The person sees me sitting at a computer, but does not notice or want to pay attention to the salad.) Many times these three perspectives collapse and there is only a single label that would be relevant, but it should be kept in mind that they do not necessarily do so. We risk over-simplifying the world and losing valuable information if we assume that they can be conflated. Instead, multimedia systems must be careful to maintain multiple views, i.e., a video that for one person (e.g., a government official) depicts a riot, might for another (e.g., a concerned citizen) depict a demonstration.

The (t, p, a, v) definition of an event is sometimes constrained by a fourth factor, namely, advanced human planning. Multimedia research that looks at planned events focuses on events that humans organize for social purposes and that can therefore be anticipated in advance of their occurrence. This group of events includes events like concerts, games, conferences and parties. It is generally referred to as "Social Event Detection".

The Social Event Detection (SED) task at the MediaEval Multimedia Benchmark started in 2011 and has been drawing a steadily increasing number of participants each year.  MediaEval SED 2013 offers the most ambitious and interesting SED task to date. The SED Task Organizers have organized workshops and special sessions at various conferences, for example, recently the Special Session on Social Events in Web Multimedia at ICMR 2013. The MediaEval bibliography includes a relatively up-to-date list of the papers that have been published regarding the MediaEval SED task.

The SED task is defined such that its multimedia aspect arises because addressing the task requires combining different information sources (text, photos, videos) from different social communities on the Web. Note that it is the use of the (t, p, a, v) definition of an event and not per se the social nature of the data that distinguishes SED from other types of event detection in multimedia.

Another important type of event detection is defined as involving not the full (t, p, a, v), but rather (v). In other words, this variant of event detection is interested not in any specific event, but in detecting the occurrence of instances of a particular event type. This type of event detection is referred to as Multimedia Event Detection and has been offered as a task in TRECVid since 2013.  Examples of these sorts of events are "Birthday party" (from TRECVid MED 2011) and "Giving directions to a location" and "Winning a race without a vehicle" (from TRECVid MED 2012).

If you consider only the labels that they use to refer to events, SED and MED look very much the same. However, it is important to remember that for MED, multimedia that is considered relevant to the event "birthday party" can depict any birthday party at any time, at any place around the world. In other words, for MED "birthday party" is an instantiation of any event of the type birthday party. Only (v) and not the full (t, p, a, v) are part of the definition. For SED, "birthday party" would be for example, my birthday party, taken on my birthday in 2013, at the particular place at which I celebrated.

Again make note of the task definition of MED. The MED task is defined such that its multimedia aspect arises because addressing the task requires combining different modalities within the same video (visual + audio channel). Typically, the data is not social video per se. Note that it is the use of the (v) definition of an event and not per se the nature of the data that distinguishes MED from other types of event detection in multimedia.

My impression is that some researchers in the community are convinced that the way forward for the research community is to first detect (v) (i.e., apply the MED event definition) and then filter the detection results to be constrained to (t, p, a, v) (i.e., generate a list of results that follows the SED definition).

Before making this assumption, I would urge researchers to carefully contemplate the use scenario of their applications. For example, if I have one picture from a birthday party I attended and I want to search for other pictures of the same birthday party on the Internet, it does not make any sense at all to solve the complete MED birthday party detection problem as the first step in the process.

As far as weddings go, if we are using visual features it's tempting to rely on the presence of that beautiful white wedding gown to detect instances of v = "wedding". However, despite an initial impression that the gown provides a stable visual indicator, its not going to get you very far in a  real-world data set:


Ultimately, we need to look at both (t, p, a, v) and (v) and all the definitions of event detection in multimedia that lie in between. Luckily there are events in which researchers with all perspectives come together. I have now finished both my lunch and my blog post and, as a final note, I leave you with an example of just such an event:

Vasileios Mezaris, Ansgar Scherp, Ramesh Jain, Mohan Kankanhalli, Huiyu Zhou, Jianguo Zhang, Liang Wang, and Zhengyou Zhang. 2011. Modeling and representing events in multimedia. In Proceedings of the 19th ACM international conference on Multimedia (MM '11). ACM, New York, NY, USA, 613-614

Discriminative Email Writing

The best email writers that I know write discriminatively. I don't me that the are discriminating in what they write or how they phrase it. Rather, I mean that when they write they try to discriminate that email that they are writing from a thousand other emails that they have also written and also from a thousand other emails that they can imagine have recently flooded the in-boxes of people that they are writing to. They try to make sure that every email is self-contained and does not contain dependencies that need to be traced through its predecessors in order to ensure full interpretation.

At the moment, our computer science bachelors students are carrying out their final presentations and my inbox is flooded with emails that all read "Hi, here is our thesis." OK. I allowed myself to exaggerate that statement just a bit for dramatic effect. It isn't, however, that gross of an exaggeration.

I decided that I would develop a list of emailing rules, and if I can get them refined enough, I will use them to help guide students on how they can most efficiently communicate with me (hoping, in the process to help them towards writing more professional emails and at the same time keeping my own email skills honed).

• Please put the name of the project that you are communicating about in the subject line of the email.
• Please put a keyword that reveals the nature of the issue in the subject line of the email. For urgent issues, you can also include the words "time sensitive".
• Please start the email with one sentence that states what you need or what you are asking. Then, if there are additional explanations, put them in the second paragraph. The supporting information is very welcome, but please do not bury the main question or request deep in the email.
• If you are referencing a past email, please include a copy of the past email in your email. 
• If you are referring to a date, please write Friday 28 June and not "next Friday". If you need to refer to a time, please repeat the relevant dates, e.g., "A week before our presentation, which is scheduled for Thursday 4 July."
• If you are referring to a URL mentioned in a past email, please repeat the URL in every future mail that needs to make the same reference.
• If you are write an email and expect that the answer will be relevant to the whole project team, please put all the members of the team on cc so that the answer can go via reply-to-all instead of forwarding them the answer afterwards. (This let's everyone involved in the communication know exactly who knows what.)

It is curious to me, because thinking discriminatively seems to come naturally to our students. The question "Which keyword query do you need to use in order to find X document that you read last week on your hard drive" is not a particularly hard one. Put differently, as humans it seems that we can think of a document,  and relatively easily come up with a word or phrase that will discriminate that document from hundreds of others we read. It would be nice if would could more consistently apply the inverse sort of thinking when we write email so that we ourselves generate documents that are very easy to discriminate.

The Five Runs Rule: Less is More in Multimedia Benchmarking

MediaEval is a multimedia benchmarking initiative that offers tasks in the area of multimedia access and retrieval that have a human or a social aspect to them. Teams sign up to participate, carry out the tasks, submit results, and then present their findings at the yearly workshop.

I get a lot of questions about something in MediaEval that is called the "five-runs rule". This blogpost is dedicated to explaining what it is, where it came from, and why we continue to respect it from year to year.

Participating teams in MediaEval develop solutions (i.e., algorithms and systems) that address MediaEval tasks. The results that they submit to a MediaEval task are the output generated by these solutions when they are applied to the task test data set. For example, this output might be a set of predicted genre labels for a set of test videos. A complete set of output generated by a particular solution is called a "run". You can think of a run as the results generated by an experimental condition. The five-run rule states that any given team can only submit of five sets of results to a MediaEval task in any given year.

The simple answer to why MediaEval tasks respect the five-runs rule is "They did it in CLEF'. CLEF is the Cross Language Evaluation Forum, now the Conference and Labs of the Evaluation Forum, cf. http://www.clef-initiative.eu/. MediaEval began as a track of CLEF in 2008 called VideoCLEF and at that time we adopted the five-runs rule and have been using it ever since.

The more interesting answer to why MediaEval tasks respect the five-runs rule is "Because it makes us better by forcing us to make choices". Basically, the five-runs rule forces participants during the process of developing their task solutions to think very carefully about what the best possible approach to the problem would be an focus their effort there. The rule encourages them to use a development set (if the task provides one) in order to inform the optimal design of their approach and select their parameters.

The five-runs rule discourages teams from "trying everything" and submitting a large number of runs, as if evaluation was a lottery. Not that we don't like playing the lottery every once in a while, however, if we choose our best solutions, rather than submitting them all, we help to avoid over-fitting new technologies that we develop to a particular data set and a particular evaluation metric. Also, if we think carefully about why we choose a certain approach when developing a solution, we will have better insight about why the solution worked or failed to work...which gives us a clearer picture of what we need to try next.

The practical advantage of the five-runs rule is that it allows the MediaEval Task Organizers to more easily distill a "main message" from all the runs that are submitted to a task in a given year: the participants have already provided a filter by submitting only the techniques that they find most interesting or predict will yield the best performance.

The five-runs rule also keeps Task Organizers from demanding too much of the participants. Many tasks discriminate between General Runs ("anything goes") and Required Runs that impose specific conditions (such as "exploit metadata" or "speech only"). The purpose of having these different types of runs is to make sure that there are a minimum number of runs submitted to the task that investigate specific opportunities and challenges presented by the data set and the task. For example, in the Placing Task, not a lot of information is to be gained from comparing metadata-only runs directly with pixel-only runs. Instead, a minimum number of teams have to look at both types of approaches in order for us to learn something about how the different modalities contribute to solving the overall task. Obviously, if there are two many required runs, participating teams will be constrained in the dimensions along which they can innovate, and that would hold us back.

Another practical advantage of the five-runs rule has arisen in the past years when tasks, led by Search and Hyperlinking and also Visual Privacy, have started carrying out post hoc analysis. Here, in order to deal with the volumes of the runs that need to be reviewed by human judges (even if we exploit crowdsourcing approaches), it is very helpful to have a small focused set of results.

Many tasks will release the ground truth for their test sets at the workshop, so teams that have generated many runs can still evaluate them. We encourage the practice of extending the two-page MediaEval working notes paper into a full paper for submission to another venue, in particular international conferences and journals. In order to do this, it is necessary to have the ground truth. Some tasks do not release the ground truth for their test sets because the test set of one year becomes the development set of the next year, and we try to keep the playing field as level as possible for new teams that are joining the benchmark (and may not have participated in previous year's editions of a task).

In the end, people generally have the experience that when they are writing up their results in their two-page working notes paper, and trying to get to the bottom of what worked well and why in your failure analysis, they are generally quite happy that they are dealing with not more than five runs.

Multimedia Readymades

This blog post is a continuation of http://ngrams.blogspot.nl/2013/03/multimedia-bits.html and a response to Cynthia Liem's call to collect "hidden gems" of internet multimedia made during her TedX Delft talk "Every bit of it" (cf. http://youtu.be/ykkXdjr0rfI?t=15m58s) Cynthia says that original bits of multimedia that we share on the Internet have the feel of "randomness" or of "low quality", but that what they are are "entry points" to things of value, if we can discover them and polish them. In this blog post I relate discovered bits of multimedia to the surrealist concept of "Readymades".

An art exhibition that I visited some 25 years ago in Frankfurt was probably what set me first to thinking about hidden gems and the process by which value is assigned to what would conventionally be considered mundane.

My most recent hidden gem is a video that captures the comments of people on something that has been created not by an act of coming-into-being, but by an act of putting-into-context.

From one perspective, the video is about Fountain, which is one of the Readymades of Marcel Duchamp. Wikipedia gives us the definition of the Readymade from André Breton and Paul Éluard's Dictionnaire abrégé du Surréalisme: "an ordinary object elevated to the dignity of a work of art by the mere choice of an artist." 

The Fountain is a urinal that has been taken out of context in two ways: first, it lies flat rather than hanging on a wall in the position it would need to be in order to be used, and, second, it was submitted by Marcel Duchamp in 1917 as a work of art to an art exhibition.

One could quip, "Beauty lies in the eye of the beholder." and let it go at that. However, scratch a little deeper, look beyond the shock-value of the object being a urinal, and a different message becomes clear: The artifact itself has little importance, and merely provides a trigger for the process of Fountain becoming something interesting and worthwhile.

Marcel Duchamp 'FOUNTAIN' - IS IT ART? from arlen figgis on Vimeo.

From another perspective, the video is about what happens when you take a widely-recognized work of art and display it to people out of context. In the video, the urinal is put into a public toilet in Liverpool and people, who recognize the urinal as Fountain, are asked to comment on it.
 
Of all the people in the video, only two comment on the fact that they are actually standing in a toilet looking at a toilet. The relative inattention to this point suggests that the fact that Fountain is finally in its "home" surroundings does not have much impact on people's opinions of it. No one in the video is looking at the urinal and having anything that at all related to a "The-Emperor-has-no-clothes" moment. Instead, they seem to have the same thoughts and feeling as they did when they walked into the toilet: they are interested, and they acknowledge it as something worthwhile.

I consider this video to be rather unfinished (yet a "diamond in the rough") because it has the power to make a point that it does not explicitly make. The un-made point is this: it is non-trivial to undo the Fountain effect, i.e., the urinal stubbornly resists returning to un-interestingness. Once a gem has emerged from a mundane object, the gem-status is difficult to shake.

These two perspectives on what this video is about represent two poles of a spectrum of interactions that we (as users, uploaders and viewers) have with bits of online multimedia that people capture and upload to the internet.

In some moments, we amuse, educate and otherwise occupy ourselves through layers of rediscovery, and the popularity, quality, origin, and perhaps even topic of the original multimedia object could not be less relevant.

In other moments, we forget about  rediscovering hidden gems and return to what it widely-acknowledged to be interesting or worthwhile, we cling to The Cannon or flock to watch the blockbusters.

Our goal as multimedia information retrieval researchers should be to develop techniques that support this entire spectrum.

In Frankfurt 25 years ago, my encounter with Readymades revealed to me that it is not particularly useful to assume that there is a sharp boundary between creation and discovery. It made clear that significance arises through the dialogue representing an interplay between a large number of voices representing the general public and more limited number of voices recognized as authorities.

These poles exist now, they existed 25 years ago (pre-Internet), it was happening in 1917, and from there it seems a relatively straightforward extension to the idea that they were there before and that their importance will continue.

It is multimedia systems that give us access to the hits and classics, but that also allow us to discover the triggers ("entry points" in Cynthia's words) from which interesting and worthwhile objects emerge and to support the interplay of contributions and opinions necessary for emergence.

It's a difficult problem. I rather think that the number of people that share my understanding of my hidden gem is quite limited, and that we would need a retrieval or recommendation system that would reach all of them in order to really add any polish. Or maybe it is enough for the gem to be personal, although I am far from sure that it is easy to ensure this video is still discoverable in future years when I again return to the topic of the emergent value of the mundane.

Multimedia Bits

Cynthia Liem recently gave a TedX Delft talk called "Every bit of it". I have the pleasure and the honor of being her colleague at the Delft Multimedia Information Retrieval Lab. I missed being at the talk in person, but finally found time today to watch it on YouTube.

What she is saying is so critical---it at the same time both fresh and timeless---that the talk deserves a more in-depth reaction than a tweet or re-tweet. This post summarizes the message that I heard in Cynthia's talk.

Cynthia makes the point that every bit on the Internet has meaning to the original person who put it there. These are of course, multimedia bits, that she is speaking of videos and images that have been captured by people and shared on the Internet. An important question in this era of Big Data: each of these "bits" of multimedia is captured by someone for some reason. Cynthia tells us that we can add worth to these bits by enhancing them with other bits; in particular, she shows us wondrous transformations that can be brought about by adding music to video.



Towards the end of the video, she says that a big question that with face with multimedia is "What is relevant?" http://youtu.be/ykkXdjr0rfI?t=14m1s

She points out that we tend to focus for the obvious in our understanding of relevance, e.g., popularity and quality, and that because of this focus, we miss material that people do not know exists.

Instead, the "bits" of multimedia on the Internet should be seen as an entrance to the world, not as the final product, but a place to start. A diamond in the rough that needs to be polished in order to add value.

At the end of the video, she asks the audience to keep their eyes open for a rediscovered hidden gem....I discuss my gem in my next post: http://ngrams.blogspot.nl/2013/03/multimedia-readymades.html

Visual Relatedness is in the Eye of the Beholder: Remember Paris

How do we know if a tag is related to the visual content of an image? In this blogpost, I am going to argue that in order to answer that question, it is first necessary to decide who "we" is. In other words, it is necessary to first define who is the person or persons who are judging visual relatedness, and only then ask the question is this tag related to the visual content of the image.

I'll start out by remarking that an alternate way of approaching the issue is to get rid of the human judge all together. For example, this paper:

Aixin Sun and Sourav S. Bhowmick. 2009. Image tag clarity: in search of visual-representative tags for social images. In Proceedings of the first SIGMM workshop on Social media (WSM '09). ACM, New York, NY, USA, 19-26.

provides us with a clearly-defined notion of the "visual representativeness" of tags. A tag is considered to be visually representative if it describes the visual content of a photo. "Sunset" and "beach" are visually representative and "Asia" and "2008" may not be. A tag is visually representative if it associated with images whose visual representations diverge from that of the overall collection. The model in this paper uses a visual clarity score, which is the Kullback-Leibler divergence of language models based on visual-bag-of-words representations.

Why don't we like this alternative? Well, this definition of visual representativeness does not reflect visual representativeness as perceived by humans. It's not clear that we really are helping ourselves build multimedia systems that serve human users if we make things less complicated by getting rid of the human judge.

The issue is the following: Humans have no problem confirming that an image depicting a pagoda at sunset and an image depicting a busy intersection with digital billboards both depict "Asia".  There is something about the visual content of these two images that is representative of "Asia", and it seems to be a simple leap from there to conclude that the tag "Asia" is related to the visual content of these images.

But there was a time in my life where I didn't know what a pagoda was. It was less long ago than one may think (although certainly before the workshop at which the paper above was presented, held at ACM Multimedia 2009 in Beijing), which prompts me to think further.

A solution might be the following: We could stipulate that in my pre-pagoda-awareness years, I should have been excluded from the set of people who gets to judge if photos are related to Asia. But then would would then have to worry about my familiarity with digital billboards, and then the next Asia indicator and on and on until I and everyone that I know is excluded from the set of people who gets to judge the visual relatedness of photos to tags. In short, this solution does not lead to a clearer definition of how we can know that a tag relates to the visual content of an image.

Why do things get so complicated? The problem, I argue, is that we ask the question of a pair: "For this image and this tag (i,t) is the visual content of the image related to this tag?"  This question does not lead to a well-defined answer.

The answer is, however, well defined if we ask the question of a triple: "For this image, this tag and this person or group of people (i,t,P): is the visual content of the image related to this tag in the judgement of this person or group of people?" In other words, we need to look for the relationship between tags and the visually depicted content of images in the eye of the beholder.

We can then perform a little computational experiment: Put person or people P in a room and expose them to the visual content of image i and ask the yes/no question "Is tag t related to image i?"

The answer of P is going to depend on the method that P uses in order to reason from the visual content of i to the relatedness of tag t. Here's a list of different Ps who are able to identify Paris for different reasons.

(i, "paris" P1): I took the picture and when I see it, I remember it.
(i, "paris" P2): I was there when the picture was taken and when I see it, I remember this moment.
(i, "paris" P3): Someone told me about a picture that was taken in Paris and there is something that I see in this picture that tells me that this must be it.
(i, "paris" P4): I know of another picture that looks just like this one and it was labeled Paris.
(i, "paris" P5): I've seen other pictures like this an recognize it (the specific buildings that appear).
(i, "paris" P6): I've been there and recognize characteristics of the place (the type of architecture).
(i, "paris" P7): I am a multimedia forensic expert and have established a chain of logic that identifies the place as Paris.

Perhaps even more are possible. What is clear is the following: It would be nice if we would have ended up with two P's: expert annotators and non-expert annotators. However, it looks like what we have is judgements that are based on quite a few differences in personal history, previous exposure, world knowledge, and expertise.

If we want to develop truly useful algorithms that validate the match between the visual content and the tag, we have a lot more work to do, in order to cover all the (i,t,P).

The key is to get a chance to question enough Ps. Multimedia research needs the Crowd.

Crowdsourcing for Multimedia: ACM Multimedia and The Crowd

Crowdsourcing for multimedia is a set of techniques that leverage human intelligence and a large number of individual contributors in order to tackle challenges in multimedia that are conventionally approached using automatic methods. Exploiting the crowd means taking advantage of human computation where it can help support multimedia algorithms and multimedia systems the most.

ACM Multimedia 2013 has introduced a new Crowdsourcing for Multimedia area:
http://acmmm13.org/submissions/call-for-papers/#crowdsourcing
The area cuts clean across traditional multimedia areas, touching upon nearly every topic relevant for multimedia. The area casts a wide net to include the full range of research results and novel ideas in multimedia that are made possible by the crowd, i.e., they exploit crowdsourcing principles and techniques.

How did this new area arise? Crowdsourcing's grand debut at ACM Multimedia is CrowdMM http://www.crowdmm.org/. On Octber 29, 2012 the CrowdMM 2012 International ACM Workshop on Crowdsourcing for Multimedia was held in conjunction with ACM Multimedia 2012 in Nara, Japan. The workshop kicked-off with a keynote entitled "PodCastle and Songle: Crowdsourcing-Based Web Services for Spoken Content Retrieval and Active Music Listening" by Masataka Goto of the National Institute of Advanced Industrial Science and Technology (AIST), Japan. These two systems dazzled the audience and gave us a foretaste of the possibilities that the power of the crowd opens for the multimedia community. An interesting day of talks, posters and discussion ensured, culminating in a panel (summarized below).

The organizers of CrowdMM 2012 hope that both the ACM Multimedia area (focused on groundbreaking research results) and also CrowdMM workshop (focused on methodology, exploratory work and on researcher interaction) will provide a solid foundation that allows crowdsourcing for multimedia to grow within the multimedia community to reach its full potential.

"Crowdsourcing for multimedia: At a crossroad or on a superhighway?"

Summary of the Panel Discussion at CrowdMM 2012 

What is the potential of crowdsourcing for ACM Multimedia?
We need The Crowd to allow us to build larger, up-to-date dictionaries for multimedia annotation. We also need The Crowd to create ground truth at a large scale.

The combining techniques for active learning and for incentivizing human contributions will contribute to many different specific multimedia problems.

In all cases, both quality control will be important and also making it fun for The Crowd to contribute, e.g., continuing to build entertaining games to collect Crowd contributions. User engagement breeds quality: for example, Songle provides services that are enjoyable to use and attracts good workers naturally. 
 
What are the limitations of crowdsourcing for multimedia?
Data from non-experts is valuable, but for some tasks we need experts. We need methods that will allow us to identify experts, for example, with domain knowledge. The multimedia community can potentially address the problem of having access to "the right crowd", by joining forces to cultivate a community of crowdsourcing workers who deliver high quality annotations for specific multimedia domains.

How would ACM Multimedia be different had crowdsourcing been invented 20 years ago? If crowdsourcing had existed 20 years ago, we would make much more effective use of active learning paradigms, i.e., algorithms that would interactively query human annotators to obtain new labels for certain multimedia items.

Crowdsourcing makes possible large scale multimedia annotations. Even if crowdsourcing existed 20 years ago, we may not have the tools and techniques to deal with large scale data.

The challenge today is to realize the potential of multimedia, both in venturing into new domains for research and also in scaling up our systems to exploit larger amounts of human labeled data for training and also for evaluation.

In short, the panel concluded, it’s up to ACMMM to catch up with the crowd.

A big thank you to my fellow organizers for the work that they put into making CrowdMM 2012 such a success: