Teaching the First Steps in Data Science: Don't Simplify Out the Essentials

Teachers of Data Science are faced with the challenge of initiating students into a new way of thinking about the world. In my case, I teach multimedia analysis, which combines elements of speech and language technology, information retrieval and computer vision. Students of Data Science learn that data mining and analysis techniques can lead to knowledge and understanding that could not be gained from conventional observation, which is limited in its scope and ability to yield unanticipated insights.

When you stand in front of an audience who is being introduced to data science for the first time, it is very tempting to play the magician. You set up the expectations of what "should be" possible, and then blow them away with a cool algorithm that does the seemingly impossible. Your audience will go home and feel that the got a lot of bang for their buck---they have witnessed a rabbit being pulled from a hat.

However: will they be better data scientists as a result?

In fact, if you produce a rabbit from a hat, your audience has not been educated at all, they have been entertained. Worse case they have been un-educated, since the success of the rabbit trick involves misdirection of attention away from the essentials.

My position is that when teaching the first steps in data science, it is important not to simplify out the essentials. Here, two points are key:

First, students must learn to judge the worth of algorithms in terms of the real-world applications that they enable. With this I do not mean to say that all science must be applied science. Rather, the point is that data science does not exist in a vacuum. Instead, the data originally came from somewhere. It is connected to something that happened in the real-world. Ultimately, the analysis of the data scientist must be relevant to that "somewhere", be it a physical phenomenon or a group of people.

Second, students must learn the limitations of the algorithms. Understanding an algorithm means also understanding what it cannot be used for, where it necessarily breaks down.

At a magic show, it would be ridiculous if a magician announced that his magic trick is oriented towards the real-world application of creating a rabbit for rabbit soup. And no magician would display alternative hats from which no rabbit could possibly be pulled. And yet, as data science teachers, this is precisely what we need to do. It is essential that our students know exactly what an algorithm is attempting to accomplish, and the conditions that cause failure.

Yesterday, was the final day of the Multimedia Information Retrieval Workshop at CCRMA at Stanford, and Steve Tjoa gave a live demo of a simple music identification algorithm. It struck me as a great example of how to teach data science. As workshop participants we saw that they algorithm is tightly connected to reality (it was identifying excerpts of pieces that he played right there in the classroom on his violin), and his demo showed its limitations (it did not always work).

This exposition did not simplify out the essentials. Students experiencing such a live demo learn the algorithm, but they also learn how to use and how to extend it.

We were blown away not so much by the cool algorithm, but by the fact that we really grasped what was going on.

Experiences like this are solid first steps for data science students, and will lead to great places.


Postscript:

That evening, one of my colleagues asked me if I still wrote on my blog. No, I said, I had a bit of writer's block. I had been trying to write a post on Jeff Dean's keynote at ACM RecSys 2014, "Large Scale Machine Learning for Predictive Tasks", and failing miserably. The keynote troubled me, and I was attempting to formulate a post that could constructively explain why. Ten months past.

With the example of Steve's live demo it became clear why my main problem was with the keynote. It contained nothing that I could demonstrate was literally wrong. It was simply a huge missed opportunity. 

Since ACM RecSys is a recommender system conference, many people in the room were being thinking about natural language processing and computer vision problems for the first time. The keynote did not connect its algorithms to the source of the data and possible applications. Afterwards, the audience was none the wiser concerning the limitations of the algorithms it discussed.

I suppose some would try to convince me that when listening to a keynote (as opposed to a lecture) I need to stop being a teacher, and go into magic watching mode, meaning that I would, suspend my disbelief.  "That sort of makes sense, it looks pretty good" Dean said to wrap up his exposition of paragraph vectors of Wikipedia articles. 

https://www.youtube.com/watch?v=Zuwf6WXgffQ&feature=youtu.be&t=4m20s

If you watch at the deep link, you see that he would like to convince us that we should be happy because the algorithm has landed music articles far away from computer science. 

In the end, I can only hope that the YouTube video of the keynote is no one's first steps in data science.

Independently of a particular application, landing music far away from computer science is also just not my kind of magic trick.

The Dress: View from the perspective of a multimedia researcher

For those of us who are active as researchers in the area of multimedia content analysis, the fury over the color of the "The Dress" drives to the heart of our scientific interests. Multimedia content analysis is the science of automatically assigning tags and description to images and videos based on techniques from signal processing, pattern recognition,  and machine learning. For years, the dominant assumption has been that the important things that people see in images on the Internet can be characterized by unambiguous descriptions, and that technology should therefore attempt to also predict unambiguous labels for images and videos.

This assumption is convenient if you are automatically predicting a description for an image, because your technology only needs to generate a single single description. Once you have predicted your description, you only need to ask one person whether or not your prediction is correct.

However, convenient is not always useful to users. As multimedia analysis gets more and more sophisticated, it's not longer necessary to go with convenient, and we can start to try to automatically describe images in the way that people see them (i.e., both the "white and gold" and the "blue and black" camps that characterize "The Dress" debate).

We recently wrote a chapter in a book on computer vision entitled "Using Crowdsourcing to Capture Complexity in Human Interpretations of Multimedia Content" [1]. Wrapped up into a single sentence, our main point was: "It's complicated" and multimedia research must embrace that complexity head on. In the chapter, we plea for research on "multimedia descriptions involving a complex interpretation". Here's how we defined it:

Multimedia description involving a complex interpretation: A description of an image or a video that is acceptable given a particular point of view. The complex interpretation is often accompanied by an explanation of the point of view. It is possible to question the description by offering an alternative explanation. It does not make sense to reference a single, conventionally accepted external authority.

Some people look at an image and see one thing, some people look at an image and see another thing, and that is normal and ok. In the case of "The Dress", the debate quickly moved from what people saw when they looked at the image, to what they saw when looking at the actual dress of which the image was taken. Looking at the image, and looking at the object can give people two different impressions. That also is normal and ok.

If you accept the opinion about the color of "The Dress" as being decided by referencing the opinion about the actual real-world dress as an external authority, then you have indeed solved the problem. Wired, for example, does this http://www.wired.com/2015/02/science-one-agrees-color-dress The New York Times compares the image to other images of the real-world dress http://www.nytimes.com/2015/02/28/business/a-simple-question-about-a-dress-and-the-world-weighs-in.html

However, from the perspective of multimedia research, the interesting point about "The Dress" is that it is not a debate about a real-world dress, but rather about an image of that dress. Not in every case in which we analyze the content of the photo, is it possible to go to find and inspect the real world object, most photos on the Internet are just photos in and of themselves, and we interpret them without direct knowledge or or connection to the real-world situation in which they were taken.

Our perceptual and cognitive lives as human beings are rich and interesting. We stretch ourselves, grow in our intellectual and emotional capacity, when we discover that not everyone see this from the same point of view. The lesson of "The Dress" for multimedia research is that we should embrace the ambiguity of images.

If we don't see how important ambiguity is to our relationship to the world and each other, we endanger the richness in our lives. Specifically, the danger is that new image search technologies, such as that used by Google, will start providing us with a single unique answer. The fury over "The Dress" illustrates that faithfulness to how people interpret images requires that there are two answers.

[1] Larson, M., Melenhorst, M., Menéndez, M. and Peng Xu. Using Crowdsourcing to Capture Complexity in Human Interpretations of Multimedia Content. In: Ionescu, B. et al. Fusion in Computer Vision – Understanding Complex Visual Content, Springer, pp. 229-269, 2014.

The making of a community survey: Contrastive conditions and critical mass in benchmarking evaluation

Each year, the MediaEval Multimedia Benchmark offers a set of challenges to the research community involving interesting new problems in multimedia. Each challenge is a task consisting of a problem description, a data set, and an evaluation metric.

The tasks are each organized independently, each by a separate group of task organizers. Each task focuses on developing solutions to very different problems. However, they are held together by the common theme of MediaEval: social and human aspects of multimedia. A task has a human aspect if it considers modeling the variation in people’s interpretations of multimedia content, including dependencies on context and intent, is not considered variability that must be controlled, but rather part of the underlying problem to be solved. A task has a social aspect if the task develops technology that supports people in developing and communicating knowledge and understanding using multimedia content.

In addition to the human and social aspects, MediaEval tasks are united by the common goal of moving forward the state of the art in multimedia research. To this end, they strive to achieve both qualitative and quantitive insight into the algorithms that are designed by participating teams to address the challenges. We can call qualitative insight "what works" and quantitative insight "how well it works".

How well an algorithm works must necessarily be measured against something. Most obviously, an algorithm works well if the people who actually have the problem that lies at the root of the task agree that the algorithm solves the problem. These people are referred the "problem holders" or "stakeholders", they are usually a company, or, very often, a set of end users of the multimedia technology. In evaluation campaigns such as MediaEval, the formulation of the problem is represented by the data set and the problem definition. Their opinion of what constitutes a solution is represented by the ground truth (i.e., the reference labels for the data set) and the evaluation metric.
In a living labs set up for algorithm evaluation, both the data set and the ground truth are streams, and move closer to actually instantiating the problem rather than representing the problem. However, we are always directly at understanding whether one algorithm can indeed be considered to give better performance than another, i.e., the state of the art.

In order to be fairly and meaningfully compared, two algorithms must represent "contrastive conditions". This means that there is one, constrained respect in which they differ from each other. If there are two or more major differences between two algorithms, then its unclear of why one performs better than the other. In real life, we might not care why, and simply choose the better performing algorithm. However, if we take the time to investigate contrastive conditions, then we can isolate "what works" from "what doesn't work" and ultimately answer questions like "Have I gotten close to the ceiling of the best possible performance that can be achieved on this challenge?", and "Which types of solutions are just not worth pursuing further?". Such questions also have a key contribution to make for algorithms used in operational settings.

Each year, MediaEval publishes a survey with a long list of questions to be answered by the community. The MediaEval survey is key in ensuring that the work of the teams participating in the challenges gives rise to contrastive conditions.

• The benchmark organizers can determine whether or not there is a minimum number of people in the research community interested in the task, who would like to participate.

• The task organizers can make contact with "core participants", teams that declare their intention to participate in the task, including submitting runs and writing the working notes paper, "no matter what". Core teams allow us to ensure that there is a critical mass for any given task, and a higher chance of contrastive conditions.

• The task organizers can determined which "required runs" that people might be interested in, and adapt the design of the task accordingly. A "required run" is an algorithm that uses certain sources of data, but that might differ in its underlying mechanisms. By deciding on required runs, the community also decides on which aspects of the task it is important to be able to investigate contrastive conditions.

The MediaEval survey is notoriously difficult to prepare. Each year, a large number of different tasks are proposed, and each task has its own particular questions. 

The descriptions of the task are quite challenging to write. MediaEval tasks are planned with a low entry threshold. This means that new groups are able to step into a task, and very easily come up to speed. In other words, the newbie teams participating in MediaEval have a fair chance with respect to teams that have participated in past years. The task descriptions must include technical depth necessary to elicit detailed information from potential participants, but they cannot be formulated in task-specific "jargon" or shorthand that MediaEval participants use among themselves.

Also, the survey must be set up in a way that people can quickly answer a great number of questions for all tasks. Although in the end teams participate in only one, or perhaps two, tasks, the design of the tasks is made better if people with a general interest in, and knowledge of, multimedia research can give their opinion and feedback on as many tasks as possible.

The MediaEval 2015 survey is about to appear. At the moment, we are at 121 questions and counting. It would take a lot less time just to make a top-down decision on which tasks to run, and how to design these tasks. However, over the years we have learned how critical the survey is: the survey input allows MediaEval tasks each year to maximize the amount of insight gained per effort invested. 

We very much appreciate everyone who participates in the survey, and helps to build a highly effective benchmark, and a productive benchmarking community.

Pick-me-up pixels: Reflections on "happy" in the new year

Yes, there's the holiday season, but a lot needs to happen during that time to make sure that 2015 goes smoothly. 

Late at night, recently, I was grinding my teeth about late reviews. I was worried about me being late in reviewing for other people, and other people being late reviewing for me. In general, I was feeling like we were all getting behind before we had even started the new year.

A colleague who knew I was fretting sent me a encouraging email with some beautiful snow pictures.  My favorite one is this one. 

The moment I clicked open the .jpgs of the pictures attached in the email, magical scenes from far away shifted my mind into a state of wonder, and then joy. 

I noticed that if I stop to reflect on how it feels, the effect of an image, a bunch of numbers representing five million pixels, is physically tangible. The experience of looking at a picture like this one delivers the same pick-me-up as a cold lemonade in the hot summer, a stunning cityscape lit up at night, the sound of waves washing over rocks, or a purring cat in my lap on a long evening. 

Goodness knows I have spent enough time reading, writing, and reviewing papers about the affective impact of multimedia, and how it can be predicted by crunching pixels. But now, looking at the photos that my colleague sent, it struck me how real that impact is. As multimedia researchers, we may not be medical doctors, but we do have the responsibility of developing technologies with the power to make people feel better.

It also hit home, that the impact goes beyond the pixels. A good part of the effect is knowing that someone realized I was glum, and also the thought that ultimately I might to have a chance to visit the place where the picture was taken.

It's interesting that the picture came via email. The effect of social multimedia doesn't require a social networking platform. Given a camera, and a display device, people will exchange pictures. The existence of Facebook helps, but is not necessary...and by similar reasoning the practice of sharing pictures will survive social networks in the form that we know them today.

I imagine that the two people in this picture have also just taken a picture of the snowy trees in the lamplight and are pausing to examine it together on a mobile device. Their exchange of thoughts might lead them to discover that they are connected by their reactions to the beauty of the experience. 

Making images us together leads us to share thoughts about our ways of seeing things that we might be otherwise tempted to disregard as irrelevant or not worth further time. Whether we are moved by our similarities, or take delight in unexpected differences in our perspectives, it is a connection that might have been missed without the mediation of a moment of collaborative picture making.

Ideally, the impact of social pixels would be a positive one without exception. The couple in the picture has captured not only pixels, but also the memory of a moment, that they will be able to relive long after the snow has melted.

But we can't know for sure. The moment may be so precious, that it is overwhelming to look back on it. Emotional overload is clearly a danger in the case of heartbreak, but even if our couple is destined to live happily-ever-after, nostalgia can be a burden. 

If no single moment is overpowering, a mass of memories might still be unbearable. The image might represent one of so many moments, that reliving them each would be an exhausting and numbing experience. 

Ultimately, as users who produce and consume multimedia content, we need systems that allow us to save and find the right content, and also the right amount of content. 

We need to be open to the possibility that maybe these systems are not intelligent systems, but rather utterly simple-minded and transparent systems that just happen to be incredibly good at supporting us, as "human users", in saving and finding the right multimedia for each other.

My pick-me up moment caused by the snow images passes quickly, and the more usual train of thoughts clicks in again: 

I start wondering about what are those things in the middle of the path. Are they air ducts? Are they bee hives? How can I find out? Will I notice them if I get there? Are my photoshop skills good enough to get rid of them? Would this make a better picture? 

And then, I am struck by the thought that my mood was lifted by the pictures, but it would really be lifted if the people I am counting on would finish reviews! Which means I should also get back to mine.

The ability of multimedia to relax and revive our inner being is subtle and fleeting. Blink and you could miss it. We feel it, but we take it for granted, and our minds quickly move to other things. We forget its role in maintaining our inner balance, and our balance with the world and each other. Without this delicate equilibrium of our affective states, we would derail....produce no more papers, invent no more cool systems.

And so for 2015, I will continue to devote effort to understanding what people see in pictures, but I aspire to also remember the power that shared pixels have to lift our spirits.

Anonymous Virtual Movie Ticket for "The Interview": Sony should fight fire with a torrent of fire

On the current crisis: Sony's next move should be to set up a system by which people can buy "Virtual Movie Theater Tickets" anonymously for "The Interview". 

The payments for these tickets would be made into a "Theater Ticket" fund. When there is enough money in that fund so that Sony can offset their loss on "The Interview", then they can just release the movie to the world via anonymous torrent. 

This way,  people can watch "The Interview", and get back to their lives, not worrying that free speech has been compromised by terrorists. The critics can pass judgement on the film's tastefulness, and the world would stay the way it was. (We liked it when video content with the power to threaten lives in democratic countries remained strictly confined to the plots of horror films---remember The Ring?) 

The idea of a "Virtual Movie Ticket" is an extension of the call to Sony to "Fight fire with fire", which they are reported to be downplaying. 

Why pay for an Anonymous Virtual Movie Theater Ticket? The impasse to setting the movie free on the world via filesharing is the precedent it would set for the movie industry. Movies and TV content require resources, sometimes substantial resources, to produce---this fact remains the bottom line. Terrorist shouldn't restrict free speech, but, not forgetting the more mundane, they shouldn't be able to influence Sony stock prices with a flick of a finger either.

Why does a Virtual Movie Ticket need to be anonymous? The reason is: The last thing that should fall into the hands of the people who hacked Sony is a list of which households which bought "Virtual Movie Theater Tickets".

The same line of reasoning applies to considerations of whether the movie should be released so that people can watch it via VOD streaming. 

If the Sony servers can be hacked, then the VOD service can be hacked, as mentioned here. Again, who is watching what is not information that should be in the hands of whoever issued the terrorist threats.

Delft University of Technology has just issued a new release of Tribler, a BitTorrent client that is not dependent on central servers. The Tribler team has been working long and hard on realizing the technology necessary for anonymity:

https://torrentfreak.com/bittorrent-anonymous-and-impossible-to-shut-down-141218/

A common reaction to this research, is that the technology is only needed by "pirates" or in countries with oppressive governments, those evading either the law, or the laying low under lawlessness. 

However, with "The Interview" crisis the media consumption behavior of free citizens has been singled out by a terrorist threat. This case drives home how seemingly innocuous information can suddenly become dangerous to individuals. 

"The Interview" case dramatically highlights that it is not enough to lead a law abiding life, but that there is a clear and present need for tools that allow all citizens to take responsibility for protecting the privacy of their own information behavior.

Whatever happens next, people who had never considered filesharing before will be forced to think seriously about whether they should be keeping their movie watching behavior anonymous for the purpose of protecting themselves.

By giving people an easy, anonymous chance to pay for a "Virtual Movie Theater Ticket" before they anonymously access "The Interview" via torrent, the current crisis could have the unexpected positive outcome. Rather than being a fiasco, it would set a precedent for a major shift in the technical and economic model of movie production, beneficial to both the studios and the consumer.

The "Virtual Movie Theater Ticket" is conceptually a small step from the movie ticket sold a the box office, but in the current situation, it would be a large and liberating game changer.

"Smart Photography" is not So Smart

This post enumerates the reasons for which "Smart Photography" technology described in this article: 
Stokman, Harro. The Future of Smart Photography. Computing Now. IEEE Computer Society. pp. 66-70, July-September 2014
is an enormously bad idea. The technology attempts to categorize images at the moment at which they were taken, and prevent certain types of images from being taken in the first place. In this post, I point out the reasons for which no analogous technology exists for text production. Then I go on to argue that "Smart Photography" constrains the ability of people to record important moments, and, critically, could hinder the ability of a witness to collect evidence during a crime. I close with an example of an image, with which I exercise my freedom of expression. 

I am writing a blog post. Let's think for a moment about what is not happening. Specifically, let's reflect on the fact that www.blogger.com is not immediately attempting to put this post into a particular topic category as I write.

One reason why it is not attempting this is that, ultimately, automatic prediction of the topic of my writing is not particularly useful to me. Automatic topic detection could misinterpret my topic, or it could completely miss the fact that I was writing on a new topic that had never been written about before. My post would get misfiled, and potentially lost.

Such a text classification technology would also need to assume that topic was indeed the appropriate type of category into which I wished to sort this blogpost. There would be no room to invent new types of categories, related to e.g., style, place, sentiment or mood.

From this example, we observe: Blogging is a couple decades old, arguably older. After all of this time, www.blogger.com keeps the responsibility of categorizing blogposts firmly with the writer. I will need to click the "Labels" icon and add the tags myself.

Here is another example: As I go through a series of Mac computers, they have gotten more sophisticated over the years. However, there is no functional "autosuggest" feature that can predict in which folder I will want to store a document or presentation while I am creating it. Because of the nature of human creativity, it just doesn't make sense to do this if the computer is going to be useful for tasks that have not yet been imagined.

Finally: YouTube auto suggests categories (presumably using my metadata), with the effect that all my videos are "Science & Technology". That helps me to find, well, exactly nothing in my uploads. Everything is labeled the same. There, again, it's clear that creation must also involve classification effort, if the end effect is to be organization. We don't create towards a pre-defined set of concepts or topics. Instead, when we create content, we also create concepts.

These three examples illustrate that the idea of classification of text at the moment of creation, has not "caught on". And, it's not because we do not yet know how to train text classifiers. Text classification technology has long been considered far ahead of computer vision. We need to acknowledge that other forces are at play.

Yet, in the face of a lack of general applications that classify text at the moment of creation, computer vision researchers are now attempting to build "Smart Photography" applications that would classify images at the moment they are taken by cameras. This contradiction should make us really sit up and think hard about the implications of "Smart Photography".

"Smart Photography" is first fascinating, and then horrifying.

It's fascinating because it's fun. If my camera decided at the moment I clicked the shutter that I was taking a picture of food, I probably would take more pictures of food. I would do it because it would be cool to see if the camera "understands" food. (Yes, I'm a multimedia geek). Also: because food as a separate category is actually built into my camera, I would feel less embarrassed about taking out my device and snapping a picture before eating.

However, this kind of behavior effectively amounts to the camera teaching me what I should be taking photos of. It will subtly channel human photographic impulses down the broad and easy road, and allow less traveled paths of expression to slowly grow over.

That's sad, but not yet horrifying. Horrifying is the following:

The "Smart Photography" camera is going to prevent its user form taking certain photos entirely.

Reading the "Smart Photography" article, the computer vision experts obviously have their hearts in the right places when they envision a camera whose shutter freezes when part of a hand appears in the frame.

However: Imagine your baby's first steps. You miss the shot because you just couldn't get your finger out of the frame in time. You would much rather have a "baby plus finger" photo and be able to treasure the moment, than have your camera freeze up on you because it "saw" a finger in front of the lens and locked the shutter.

It goes on: You miss the shot of your kid scoring that amazing soccer goal, of that rare bird that you saw on your walk in the woods. You miss the shot of the damage that was done to your car in an accident because you just couldn't hold the camera perfectly correctly. A bad shot would have been better than none.

And it gets worse. The camera aspires to block adult content, making it impossible to take a picture of any scene that it classifies as pornographic. It sounds like a miracle for law enforcement the first time you hear it. But the price is too high: Basically, if the camera blocks adult content, it means that if I witness a rape, I have no way of taking a picture of it. The possibility of identifying the perpetrator is blocked by the camera itself.

Effectively, the innovation of "Smart Photography" is making possible a camera that does not work.

Returning to the comparison to the text case. What if www.blogger.com was preventing me from writing this column, as soon as it sensed that its topic included rape? Our technology does not prevent the generation of text, and we need to remain consistent with the values that tell us that lead us to the conclusion that it should not. It is a bad idea to introduce technologies that prevent the generation of images.

One important reason why our technology does not censor text on creation, is it is not people who design technology that get to make the decision about what I can and cannot express. Rather it is the legal system, which is in turn based on the values of the community at large. This system, imperfect and slow moving as it might be, represents individual citizens equally, and can be influenced by them, in the way that a technology cannot be influenced equally by everyone.

The "Smart Photography" article argues that photocopiers prevent people from copying money, i.e., paper bills, and that this technology represents a next step. The reason why money works at all is that there is a system and a society working to make sure that its purpose is unambiguously interpreted. Money is a conventionalized sign at the basis of the society, it exists at all exactly because it is not open for interpretation. Plainly stated, the argument that technology blocking photographic capture of adult content is a natural extension of blocking photocopies of money, relies on an unsound analogy, and must be discounted for this reason.

What's the alternative to "Smart Photography"?

The solution is not making photos smarter, but rather it's changing people. It's relentlessly pursuing our efforts to support each other in our communities, and to help each other make better decisions. It's about the unending quest, that begins again with each new generation: to make people smarter.

We need to assure adequate funding to the people who dedicate their careers to fighting crime. Finding perpetrators of sexual abuse/sex crimes is simply a hard task that requires a huge investment: sick minds are sick, and they will not let a new camera technology stand in the way of their evil business. With this "Smart Photography" camera, sex offenders will be incentivized to start taking pictures that are not so easy to automatically identify, and they may be able to wipe out their own footprints. There are no easy technical shortcuts that will eliminate the need for old fashion crime fighting, yes, also of the gumshoe variety.

We need to educate people. Bear selfies are stupid. Getting people to stop taking bear selfies is not a matter of creating a camera that recognizes a bear selfie situation, and blocks the shutter when someone tries to take a bear selfie. The bear selfie is a symptom of an underlying lack of reflection. It is the underlying problem, and not its superficial manifestation that needs to be addressed. The answer is about taking the time to really talk to our children, and to each other, about what is appropriate and what is not appropriate in a given situation.

Below is the most repugnant photo that I have ever posted online, but today for the first time in my life, I did not take for granted that my camera includes a functionality that allows me to take it.

MediaEval 2014 Placing Task Technical Retreat

At the end of MediaEval Workshop, the 2014 Placing Task had a technical retreat where the details of the year's crop of algorithms was assessed, and plans for the future were discussed.

I missed the first part of the meeting, because I was still doing some organizational stuff, and also saying goodbye to people (I find that so difficult to do, and I certainly didn't manage to say goodbye to everyone). However, I did take some notes on the parts that I attended and I am putting them here for posterity. 

Of course expect the usual attentional bias in what I chose to write down---possibly also in the categories that I put the notes into as well.

Moving beyond geo-location estimation

Can we formulate a data analysis task that moves beyond geo-prediction?

Can we drive the benchmark to get the task participants to uncover the weaknesses in current placing systems. What mistakes are you making, are why are you making them?

Geo-relevance

• For which images is geo-location relevant?
• Which is the location for which it is relevant?
• What is the tolerance for error? (depends on humans, applications)

Placability

In the past two years placability has been offered as part of the task, but has been disappointingly unpopular. This seems to be a matter of people not having time. We shouldn’t take the evidence as meaning that people don’t want to do it.

Alternate form of placability:

“Select a set of x images (e.g., 100 images) from the test set that you are sure that you have placed correctly and visualize them in a map"

How to support the participants

Can we release a baseline system?

Estimates for the error?

How to move beyond co-ordinate estimation

• Can we make the Placing Task more clearly application oriented?
• Are there use scenarios beyond Flickr?
• Is anyone interested in the task of Geo-Cloaking? 
• Can the task pit two teams  against each other, one cloaking and one placing?

Evaluation metric

• We think that geodesic distance is convenient, but has limits, since it doesn’t reflect the usefulness of predictions for humans within use scenarios.
• Maybe move to administrative districts
• Other metrics motivated by human image interpretation?

Ground truth

We can measure placing performance only within the error of the ground truth (cf. [2]). What can we do to work around this limitation?

• Correspondence between geo-tags and exif metadata is indicative of whether the tag is correct. See also cool new work on timestamps [4].
• Are their other easily measurable characteristics of images online that can be used to identify images/videos with reliable geo-tags at a large scale?
• Collect more human labeled data. Do we really need to have a 500,000 item size data set?

How People Judge Place

Users (i.e., humans judging images) have different ways of knowing where a picture was taken. 

It depends on the relationship between the human judging, the image, the act of image creation.

The most basic contrast is between the case in which the human judge is the photographer, and the case in which the human judge is not the photographer and also shares no life experiences with the photographer.

Previously I discussed these different relationships in a post entitled “Visual Relatedness is in the Eye of the Beholder” and also in [3].

Why is this important? Some mistakes that are made by automatic geo-location prediction algorithms are disturbing to users, some are not. Whether or not a mistake is disturbing to a particular human judge is related to the way in which the human judge knows where the picture was taken. In other words, I may “forgive” an automatic geo-location estimation algorithm for interchanging the location of two rock faces of the same mountain, unless one of them happens to be the rock face that I myself managed to scale. How people judge place, is closely related to the types of evaluation metrics we need to choose to make the Placing Task as useful as possible.

In the Man vs. Machine paper [1] sets up a protocol that gathers human judgements in a way that controls the way in which people “know” or are allowed to come to know the location of images. More work should be explicitly aware of these factors.

Embrace the messiness

The overall conclusion: anything that we can do to move the task away from "number chasing” towards insight is helpful. This means finding concrete ways to embrace the fact that the task is inherently messy.

Thank you!

Thank you to the organizers of Placing 2014 for their efforts this year. We look forward to a great task again next year.

References

[1] Jaeyoung Choi, Howard Lei, Venkatesan Ekambaram, Pascal Kelm, Luke Gottlieb, Thomas Sikora, Kannan Ramchandran, and Gerald Friedland. 2013. Human vs machine: establishing a human baseline for multimodal location estimation. In Proceedings of the 21st ACM international conference on Multimedia (MM '13). ACM, New York, NY, USA, 867-876. 

http://www.icsi.berkeley.edu/~jaeyoung/papers/mm042-choi.pdf

[2] Claudia Hauff. 2013. A study on the accuracy of Flickr's geotag data. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval (SIGIR '13). ACM, New York, NY, USA, 1037-1040.

[3] M. Larson, P. Kelm, A. Rae, C. Hauff, B. Thomee, M. Trevisiol, J. Choi, O. van Laere, S. Schockaert, G. J. F. Jones, P. Serdyukov, V. Murdock, and G. Friedland. The benchmark as a research catalyst: Charting the progress of geo-prediction for social multimedia. In J. Choi and G. Friedland, editors, Multimodal Location Estimation of Videos and Images. Springer, 2015.

[4] Thomee, B., Moreno, J.G.,, Shamma, D.A. Who’s Time Is it Anyway? Investigating the Accuracy of Camera Timestamps. ACM MM 2014, to appear. http://www.liacs.nl/~bthomee/assets/14time_p.pdf