The MediaEval Workshop: What it's meant to be and why you want to be there.

The MediaEval workshop is the event held each year in the fall at the culmination of the yearly benchmarking cycle. At the time of the MediaEval workshop many things have already happened in the benchmarking year: The task organizers have worked hard to define tasks and issue data sets. Participants have worked hard to develop algorithms that tackle the tasks, and they have run these algorithms on the data sets. The "runs" have been evaluated and each participating team has written their working notes paper. Now it's time for the workshop!

This blog post provides a view (from my perspective as one of the MediaEval co-ordinators) on the history of the workshop and on what the workshop is meant to be. In particular, it highlights the similarities and differences with other types of workshops.

The main goal of the MediaEval workshop is to bring everyone who carried MediaEval tasks together in one physical location to present and discuss their results, exchange experiences and develop ideas for how to improve their algorithms. The first year that we met at a medieval convent, Santa Croce in Fossabanda, it was mostly due to delight in the wordplay between medieval and MediaEval. However, we soon came to appreciate how getting everyone together working, eating and basically living in the same space creates an extremely productive focus on our our common tasks and goals. (Although unlike the nuns of the Middle Ages we don't go dashing off to prayer when we hear the convent bell.)

In 2010, we held the MediaEval workshop just before ACM Multimedia 2010, which was held at the Palazzo dei Congressi in Firenze, Italy. Santa Croce in Fossabanda is located in Pisa, about an hour's train ride from Firenze. We chose the dates and place to cut down on travel time and cost for people who wanted to attend both ACM Multimedia 2010 and the MediaEval 2010 workshop.

The next year, Interspeech 2011 came to the Palazzo dei Congressi in Firenze about the same time of year. MediaEval submitted a proposal and was granted the status of an "Official Satellite Event of Interspeech 2011". Now, instead of just taken advantage of the convenience for travel, we began emphasize hook-up to the topic of the conference: being associated with Interspeech reinforced the use of speech within MediaEval and helped us to better realize the goals expressed in the MediaEval slogan The "multi" in multimedia.

This year, the 12th European Conference on Computer Vision (ECCV 2012) will be held in Firenze. This conference provides us with the opportunity to reinforce the use of visual content within MediaEval, another of the multimedia "multi's". The MediaEval 2012 Workshop will be held right before this conference starts, again in Santa Croce in Fossabanda in Pisa. A very close contending idea for the MediaEval 2012 workshop was to hold it near 13th International Society for Music Information Retrieval Conference (ISMIR 2012) in Porto. However, the results of the MediaEval 2012 survey showed that the majority preferred to co-ordinate the date and place with ECCV 2012 and stay for a third year in Pisa.

In sum, the idea of being close to a large conference related to MediaEval topics has grown from being a convenience to being an aspect that strengthens and enriches both the benchmark and the workshop.

What exactly happens at the MediaEval workshop? The workshop consists of a series of sessions on the individual tasks. The task organizers present the task as a whole and each team presents its individual results, after which the floor is opened for discussion. More discussion and exchange occurs during meals and breaks in ad hoc groups. We try to build in a lot of space for discussion into the workshop schedule and especially try to create opportunities for students to discuss with more experienced researchers who help them to guide their efforts along the most effective path.

The proceedings are an important aspect of the workshop. The MediaEval workshop proceedings is a "Working Notes Proceedings" and consists of short (two page) papers written by the participating teams to report their results. These papers describe the algorithms that are used and present the results. They then also seek to understand the algorithm, and participants are requested to report:

• which cases are easy/difficult and why
• which approaches work best and why
• which approaches do not work well

The working notes submissions are reviewed by the task organizers. The task organizers may either accept the submission as is, or may come back to the participant team with a request for revisions of the paper. The preferred mechanism is to have the papers revised rather than to reject them --- sometimes this revision cycle means that the working notes proceedings is not ready until just before the workshop. For this reason, the working notes proceedings is distributed at the workshop on a memory stick.

After the workshop, the working notes is made available online. The 2011 the "Working Notes Proceedings of the MediaEval 2011 Workshop" was published here: http://ceur-ws.org/Vol-807/ and in the future we would like to continue to use http://ceur-ws.org/. By publishing in this way, the copyright for the individual papers is with the papers' authors.

MediaEval working notes papers are intended to be first versions of work that is later extended by the authors and submitted to mainstream venues, such as conferences and journals. The fact that the MediaEval workshop proceedings consists of short working notes and that the copyright does stays with the authors keep the proceedings consistent with the goal of reporting an initial research result, which will then be refined and extended using input from the discussion at the MediaEval workshop.

Another important goal of the workshop is to discuss the tasks themselves. Did they help to move the state-of-the-art forward? Should we improve or replace them next year? Are their new questions that need to be answered that require new tasks. Any one attending the workshop is welcome to stand up in the final session and "pitch" an idea for a new task. Tasks which receive good community support in this session have a good chance of receiving the response levels they need on the yearly MediaEval survey to run as tasks in the next year.

Finally, the workshop also aims to connect ourselves and our research to the larger community. We welcome participants from industry who have tasks that they might want to pitch to the community. We also welcome representatives from other benchmarks: MediaEval grows stronger by staying in close contact with groups running benchmarking activities in areas beyond the MediaEval core domain of human and social multimedia. MediaEval 2011 was presented at CLEF 2011, NTCIR 2011 and FIRE 2011 and in 2012 we hope to convince some of our sister benchmarks to give a reciprocal presentation on their own experiences.

As part of staying connected to the larger community, the MediaEval 2011 workshop included a poster and demo session where projects that help to organize MediaEval tasks could present their results and where industry people could make a presentation of new and interesting problems that they would like to make known to the MediaEval community as possible benchmarking tasks.

The workshop closes with a gathering of task organizers and others who have invested time and effort into the community or want to get more closely involved in the future. During this session, we reflect back on what happened so far in the benchmarking year and also discuss the MediaEval related activities that we organize beyond the core benchmarking activities. These include joint papers among all the participants of a task and also special sessions at conferences. Looking forward we also plan to get involved in organizing more workshops (of the traditional variety) at conferences and also think about the possibilities of special issues. Finally, we reflect on our MediaEval goal, To offer the community innovative new tasks related to the human and social aspects of multimedia and our slogan The "multi" in multimedia (it not necessary to be able to say the slogan with a straight face.) On the basis of these reflections, we consider where we would like to go with the benchmark in the coming year.

So why do you want to be at the MediaEval workshop? Well, if you are a task participant, it gives you an opportunity to exchange with other people working on topics similar to yours and helps you to understand and improve the algorithms that you are using to approach your task. MediaEval needs a central group of dedicated researchers to organize the tasks that make the benchmark run. Attending the workshop is a good first step to getting more deeply involved in MediaEval, for example, by proposing a task for the year.

On the MediaEval 2012 survey, we had a question concerning what the community thinks about how MediaEval should grow. I personally, want to keep the workshop as small and intimate as possible. In 2011, we had nearly 60 people and that appeared to me to be a good maximum size for the workshop. However, when I examined the survey results, I realize that I am in the minority here, and that most of the people in the community would like growth. As a result, I am changing my opinion and we will not attempt to artificially restrict growth, as long as it is sustainable.

The issues and ideas around growth is just one area in which the concept of the MediaEval workshop is evolving. One of MediaEval's strengths is that it develops from year to year, guided by the input of the community --- and in particular those people who invest the most hours of their time to make it work. I look forward witnessing and being involved in this development in the 2012 season and, we hope, to seasons beyond.

For further impressions of the MediaEval workshop, check out the MediaEval 2011 workshop video:

Core Participants in MediaEval Tasks

This week, I have been answering email from people who have proposed tasks for the MediaEval 2012 multimedia benchmark or have been contacted about becoming a core participant for a task. The emails have been asking for clarification on what exactly a MediaEval task core participant is. Instead of writing the same email back to everyone, I am putting my answer in this blog post.

The only official definition of a core participant is "A participating team that agrees, before the official beginning of MediaEval task sign up, to cross the finish line for a given task come hell or high water". I am actively looking for a paraphrase of this definition, but for the moment, please be tolerant of the use of an idiomatic phrase (and one containing the word "hell" at that.)

How can a core participant team be sure in February that they will complete a task in late summer? Usually, a core participant will be able to reliably predict that they will cross the finish line on a task because have successfully completed similar tasks in the past. In most cases, they will have a specific person or persons on their team who is working on the task as an integral part of their research project.

If you don't think that you can commit to being a core participant, that is absolutely fine. You can sign up for the task with the normal sign up procedure and participate as a general participant. General participants must make a genuine effort to complete the task and we also expect them to attend the MediaEval workshop if at all humanly possible. General participants also have a much longer window in which to consider task participation before deciding to sign up. Regular sign up won't close until the end of April 2012.

Within MediaEval, fine-grained decisions are left as much as possible to the organizers of the individual tasks. The consequences are that, other than the "come hell of high water" definition, it is left up to the task organizers to interpret exactly what a core participant is. For this reason, the detailed discussion of the matter appears on my blog and not on the MediaEval website.

In order to understand what a core participant might do in any given task, here is the story of the history of "core participants" in MediaEval tasks.

We introduced the notion of core participants after the MediaEval 2010 workshop. At the workshop, we had noticed that (not surprisingly) tasks for which there were a lot of results to compare gave us more interesting insight. We wanted to have a way of ensuring that a minimal number of teams would complete any given task, but we didn't want to use a inelegant, alienating solution, such as cancelling tasks which don't receive a minimum number of successful run submissions. For this reason, a task is required to produce evidence that it will have a minimum number of results to compare at the workshop before being officially accepted as a MediaEval task. This evidence takes the form of collecting a list of core participants.

While the season is running, the task organizers might ask the core participants to help out with the task. For example, last year core participants helped debug the evaluation script. Another example is supporting newcomers. Sometimes new participants in MediaEval need help in understanding some details of the instructions. In this case, a task organizer might ask an experienced core participant to communicate with a newbie. Newbies are often students that have just started their MSc or PhD programs and have a lot to think about at once---we don't want them getting stuck on easily answered questions. It might also save them time if they can communicate with someone in a language that they can type more quickly then English. That someone could be a core participant.

There is a clear line between a task organizer and a core participant. A core participant does not have any official responsibility other than finishing the task. Core participants help the task organizers only if they feel inspired to---it's not required. We also assume that core participants are removed enough from the data set creation process that it is impossible for them to have a relative advantage on any given task over a general participant. For this reason, their scores can be included in the official ranking of scores for a task presented at the MediaEval workshop---task organizers are highly encouraged to carry out their own tasks, but their scores are officially excluded in the ranking presented at the workshop.

The MediaEval Model for Benchmarking Evaluation

Currently, I'm working to put together the survey for MediaEval 2012. This survey will be used to decide on the tasks that will run in 2012 and also will help to gather information that we will use to further refine the MediaEval model: the set of values and organizational principles that we use to run the benchmark.

At the workshop, someone came up to me and mentioned that he had made use of the model in a different setting, 'I hope you don't mind', he said. Mind? No. Quite to the contrary. We hope that other benchmarking initiatives pick up the MediaEval model and elements of it and put them to use.

I have resolved to be more articulate about exactly what the MediaEval model is. There's no secret sauce or anything -- it's just a set of points that are important to us and that we try to observe as a community.

The MediaEval Model
The MediaEval model for benchmarking evaluation is an evolution of the classic model for an information retrieval benchmarking initiative (used by TREC, CLEF, TRECVid). It runs on a yearly cycle and culminates with a workshop where participants gather to discuss their results and plan future work.

The MediaEval attempts to push beyond existing practice, by maximizing the community involvement in the benchmark. Practically, we do this by emphasizing the following points:

1. Tasks are chosen using a survey, which gathers the opinion of the largest possible number of community members and potential MediaEval participants for the upcoming year.
2. Tasks follow the same overall schedule, but individual tasks are otherwise very autonomous and are managed by the Task Organizers.
3. The Tasks Organizers are encouraged to submit runs to their own tasks, but these runs do not count in the official ranking.
4. The Task Organizers are supported by a group of five core participants who pledge to complete the task "come hell or high water".
5. Each task has an official quantitative evaluation metric, which is used to rank the algorithms of the participating teams. The task also, however, promotes qualitative measures of algorithm goodness: i.e., the extent to which an algorithm embodies a creative and promising extension of the state of the art. These qualitative measures are recognized informally by awarding a set of prizes.

In interview footage from the MediaEval 2012 workshop, I discuss the challenge of forging consensus within the community.



One of the important parts of consensus building is collecting detailed and high-coverage information at the beginning of the year about what everyone in the community (and also potential members of the community) thinks. And so I am working here, going through not only the tasks proposals, but also other forms of feedback we've gotten from last year (questionnaires, emails) in order to make sure that we get the appropriate questions on the survey.

It always takes so much longer than I predict -- but it's definitely worth the effort.