From the Rialto: On the Relevance of Geotags

This weekend I am in Venice in between an Networked Media and Search Systems Concertation meeting and ICMR 2011. I am in the processing of preparing the slides for a talk [1] that I will give on Tuesday morning on tagging and geotagging. Once again, I find myself busy with the question of what does it mean for a geotag to be relevant to a photo.

At first blush, the answer to this question is obvious: A geotag is relevant to a photo if it gives the exact longitude and latitude of the position of the camera when the photo was taken. I have this information for the photo above, which was taken earlier today -- but it reflects so little of the larger story. On the one hand our cameras (in this case we're using a new Android phone) pin us down and on the other they tell not much at all.

I originally asked this question about the relevance of geotags after the talk of Pavel Serdyukov at SIGIR 2009, "Placing flickr photos on a map." I garbled the formulation, Pavel didn't understand it, and I have since come to understand the large number of dimensions involved in the answer. "Where does a photo truly belong on a map?" isn't an obvious question and doesn't have an obvious answer, either.

It's generally accepted that knowing the position of the photographer underspecifies the location of the photo since it is not known in which direction the camera was pointing or on what it was focused. It could possibly be the case that the subject of the picture is quite far away from the photographer, making the geo-coordinates of the photographer not the most relevant position to the photo in terms of the depicted content.

The photo from today is a good example. It was taken in Venice, but it is also relevant to Trento, since I am here in Italy on a trip to Trento and only happened to come to Venice for the weekend. I will remember the trip as a "Trento" trip. But the Trento association goes beyond my own personal sense of the proper place of the photo -- anyone who was on a business trip to Trento might find Venice related since they are possibly flying into a Venice airport or like myself also planning a touristic stay in Venice.

The sign in the background says "Per Rialto" and is pointing the way towards the Rialto bridge. We're on our way to the bridge and have taken the photo along the way -- it is relevant to the geocoordinates of the Rialto bridge in the sense that this was the final destination of the walk.

The photo was taken for my family back home. My great uncle is sick and in the hospital at this moment. He is a central node in my extended family and ties us together across space and also across time -- and for years has passed along to us the lore of the generations that preceded us -- a mixture of stories, poetry and family values -- a nourishing flow that sustains us and binds us together. One of the things that he does is write email newsletters to the family members and this newsletter has a section that is called "From the Rialto", which describes the current activities of various family members. Here I am sending my own thoughts out from the Rialto to my family back home and my great uncle in the hospital. The picture has more to do with them and where they are than here.

Further, I don't really remember why the section of the newsletter is called "From the Rialto". It might not be the Rialto in Venice (which is actually an entire district and not just a bridge). It might be an entirely different Rialto. My picture is actually related to the original Rialto used by my uncle, except that I don't know where that is. I will admit that there are probable not a lot of photos out there that enjoy this sort of relationship to place -- but the main point is the existence of possibly relevant places that the person who takes the photo is not necessarily aware.

Actually, I just wanted a picture with a Rialto sign in the background. These signs are numerous in Venice...in fact they sell T-shirts on which they are depicted. In a certain way, the photo is a "Woman in from of Rialto sign" picture and is also relevant to anywhere else a Rialto sign is located. In a way, I am standing at place that has many different geo-coordinates.

The renown of the Rialto bridge is attested by the numerous replicas that exist, for example, in Las Vegas. The original and its imitators are related -- photos of one are of interest to people visiting the location of the other. It would be useful for a picture of the Las Vegas Rialto bridge to bear a tag that relates it to the one here in Venice.

I briefly entertained the idea that it is generally accepted that the relevant geocoordinates for a photo are the ones at which the photographer is standing, since those geocoordinates would be considered relevant to the widest range of people, i.e., be the most objective. Indeed, this is the relevance criteria for geotags that we apply within the MediaEval Placing Task. My association of the photo with the folks back home is relevant to a handful of souls, whereas anyone could recognize the connection with this corner of Venice. However, replicas are part of a common shared culture across the world and the connection between an image of a replica and the geo-coordinates of the original could be considered of possible interest to anyone and not just a personal circle.

Finally, there is the question of the relevant geographical resolution. For me, this is a "Venice" picture. I don't need to know the exact corner again -- nor will I try to find it. We overheard a group of Americans talking about their vacation -- and noticed that they referred to themselves not as being in Venice or in Italy, but in Europe. Europeans hardly ever reflect upon the fact that they are in Europe...they take it for granted.

Wandering through Venice its tempting to just fall into people watching -- making up small stories about couples out enjoying the good weather and the beautiful city: how they met, how they fell in love and what the future has in store for their mutual happiness. The relevance of the geo-coordinates of the photographer is obvious for those who want to take people watching to the next level -- if the couple standing next to us a bridge near San Marco uploads their pictures to Flickr, we will know even then more about them...we could check if their trip to Venice provided the solid basis for a stable relationship, or whether they ended up parting ways. Taking the people watching to the extreme: It would finally be possible to check whether kissing under the Bridge of Sighs does its work in guaranteeing everlasting bliss.

[1] Larson, M., Soleymani, M., Serdyukov, P., Rudinac, S., Wartena, C., Murdock, V., Friedland, G., Ordelman, R. and Jones, G.J.F. Automatic Tagging and Geotagging in Video Collections and Communities. ACM International Conference on Multimedia Retrieval 2011.

ACM Multimedia SSCS 2010 Workshop on Searching Spontaneous Conversational Speech

The Fourth Workshop on Searching Spontaneous Conversational Speech took place on 29 October 2010 at ACM Multimedia. Papers were presented about techniques for speech retrieval, speaker role recognition, spoken term detection and concept detection. Invited speakers addressed challenges for the future of spoken content retrieval, including interview data, multimedia archives and the Spoken Web. The demonstrations were a highlight of the workshop. These were first introduced in a boaster session and then presented to workshop participants in an interactive session. Here's the Wordle Word cloud made from the title and the abstracts of all the papers presented!

Currently, we are getting ready for an upcoming special issue on searching speech in ACM Transactions on Information Systems.

MediaEval 2010 Workshop Report

We were delighted that Bill Bowles attended the MediaEval 2010 workshop and that he made us our own MediaEval video trailer, in which he tells the story of MediaEval from his own point of view. The MediaEval 2010 Affect Task was devoted to analyzing Bill's travelogue video from his Travel Project and ranking it by how boring viewers reported it to be. As a filmmaker, another rational reaction would be "Who are these people, what did they do to my video? I don't want to get anywhere near them!" But instead, he came, participated and told us about ourselves using the very same medium we devote so much effort to studying.



I was amazed at how quickly this video accumulated views, it quickly outstripped any video I've ever posted to the Internet. However, if video is not your thing and you want the text version of what happend here is the text of a workshop report written for a project newsletter.

MediaEval 2010 Workshop Report

The MediaEval 2010 workshop was held on Sunday, October 24, 2010 in Pisa, Italy at Santa Croce in Fossabanda. MediaEval is a benchmarking initiative for multimedia retrieval, focusing on speech, language and contextual aspects of multimedia (geographical and social context) and their combination with visual features. Its central sponsor is the PetaMedia Network of Excellence. In total, four tasks were run during MediaEval 2010. To approach the tasks, participants could make use of spoken, visual, and audio content as well as accompanying metadata. Two “Tagging Tasks’ (a version for professional content and one for Internet video) required participants to automatically predict the tags that humans assign to video content. An ‘Affect Task’ involved automatic prediction of viewer-reported boredom for Travelogue video. Finally, a ‘Placing Task’ required participants to automatically predict the geo-coordinates of Flickr video. The Placing Task was co-organized by PetaMedia and Glocal. It was also given special mention in the talk of Gerald Friedland entitled “Multimodal Location Estimation” in the “Brave New Ideas” session at ACM Multimedia 2010.

During the MediaEval 2010 workshop, researchers presented and discussed the algorithms developed and the results achieved on the MediaEval 2010 tasks. The workshop drew 29 participants from 3 continents. More information about the 2010 results including participants’ short working notes papers, are available at: http://www.multimediaeval.org/mediaeval2010
Currently, MediaEval 2010 participants are working towards a special session at the 2010 ACM International Conference on Multimedia Retrieval (ICMR 2010), which will be dedicated to presenting extended results on MediaEval 2010 tasks.

Mediaeval 2011 will be organized again with sponsorship from PetaMedia and in collaboration with other projects from the Media Search Cluster. The task offering in 2011 will be decided on the basis of participants' interest, assessed, as last year, via a survey. At this time, we anticipate that we will run a Tagging Task and a Placing Task as well as a couple innovative other, new tasks as dictated by popularity. If you are interested in participating in MediaEval 2011 or if your project would like to organize a task, please contact Martha Larson m.a.larson@tudelft.nl Additional information on MediaEval 2011 is available on the website: http://www.multimediaeval.org

MediaEval Tagging Task Professional

DIXIT, a Dutch-language journal for speech and language technology, invited me to do a piece on the "Tagging Task Professional", one of the four multimedia indexing and retrieval tasks that the MediaEval benchmarking initiative ran in 2010. I am posting an English version of the text here on my blog. The piece will appear in December, after the MediaEval 2010 workshop in October (I note that in order to explain the past tense used to describe an event that has not happend yet).

The workshop will be held in a medieval convent called Santa Croce de Fossabanda, located in Pisa, Italy. The photo here is from Flickr user Marius B, licensed under Creative Commons License by-nc-sa. I notice that I do well with attribution if I am going to print material (brochures etc.), but I get sloppy with Power Point. If I know this photo is on my blog, I will be able to mind myself it comes from Marius B quickly in case I want it in future presentations.

Many Minds Make Light Work: Bringing Researchers Together to Work towards Automatic Indexing for Cultural Heritage Multimedia Collections

"Medieval", "mediaeval" and "MediaEval" are all pronounced the same. While "medieval" and "mediaeval" are alternate spellings for a adjective describing something that occurred in the Middle Ages, "MediaEval" is a benchmark initiative that brings researchers together to tackle challenging tasks in the area of multimedia indexing and retrieval. In 2010, a group of researchers worked individually and then met at a medieval convent "Santa Croce in Fossabanda" in Italy. Can a group of MediaEval scientists solve today's challenges of automatic generation of metadata for cultural heritage multimedia content?

Cultural heritage content often takes the form of multimedia and in particular of audio and video recordings. Cultural heritage collections are often staggering in size. The archive of the Netherlands Institute for Sound and Vision houses a breathtaking 250,000 hours of video content and receives and additional 8,000 hours of content broadcast by national broadcasting companies each year. Material that is stored in such a huge collection, but is not adequately annotated, is useless since it can no longer be found by people who wish to view, reuse or otherwise study it. Professional archivists have developed a set of techniques for annotating material with metadata for storage in the archive that will ensure that it can later be found. These techniques have stood the test of time and will continue to be critical for finding multimedia content in large archives in the future. The ability to generate high quality metadata, however, is not enough. Rather, metadata production must be scaled so that incoming material can be appropriately annotated at the rate at which it arrives.

Techniques from the area of Speech and Language Technology hold promise to support archivists in the generation of archival metadata. Here, we specifically look at the problem of generating subject labels (or "keywords") for television broadcasts. Subject labels are terms drawn from the archive thesaurus. Examples of keywords are, Archeology (archeologie), Architecture (architectuur), Chemistry (chemie), Dance (dansen), Film (film), History (geschiedenis), Music (muziek), Paintings (schilderijen), Scientific research (wetenschappelijk onderzoek) and Visual arts (beeldende kunst). Automatic generation of subject labels can help archivists in one of two ways: by providing a list of suggested subject labels for a video, thus narrowing their field or choice, or, by automatically generating a best guess in order to label material which would otherwise go un-annotated due to huge volume of incoming video material and the time constraints of the archive staff.

Automatic generation of subject labels is accomplished by algorithms that make use of several data sources: production metadata for broadcasts, transcripts of the spoken content of broadcasts produced by automatic speech recognition technologies and analysis of the visual content of the broadcast recording. The algorithms apply statistical techniques including word-counts and co-occurrences and also machine learning methods. Current algorithms are, however, far from perfect and their further improvement requires sustained and concerted effort on the part of research scientists.

Many researchers are interested in working on the problem of automatically generating subject labels for cultural heritage material. However, in order for a researcher to begin working in this area, a number of problems must be faced.

1. It is necessary to have an understanding of the problem -- requires a general knowledge of how subject labels are produced in the archive and what they are used for
2. It is necessary to have access to a large amount of example data in order to develop and train algorithms
3. It is necessary to have access to data sources such a speech recognition transcripts or visual features. In general, it is not possible to generate these resources in a lab that is not already specialized in these areas
4. It is necessary to understand the work that has previously been carried out in the area in order not to duplicate techniques that have already been tried by other researchers
5. It is necessary to know how well one's own algorithms compare to the current state of the art.

The purpose of a benchmarking initiative is to address these problems and let researchers concentrate their energy on the hard work and creative thinking that it takes to develop new algorithms for important tasks. MediaEval is one of several benchmarking initiatives that pursue this paradigm. The special topic area addressed by MediaEval is multimedia, with a focus on on speech, language and social features and how they can be combined with visual features.

MediaEval promotes research progress in the area of automatically generating subject labels for cultural heritage material by running a "Task" devoted to subject labeling for professional archives. A Task is comprised of three parts: a description of the problem, a data set and a set of resources that can be used to solve the problem. Having the problem packaged as a task gives researchers easy entry to understanding the issue from the perspective of the archives and allows licensing of the data from the archive to occur in a streamlined manner. The University of Twente supplies speech recognition transcripts makes it possible for research groups without competence in Dutch-language speech recognition to contribute to developing improved approach to the task. Information about the other tasks offered can be found on the MediaEval website: http://www.multimediaeval.org/

Researchers approach the tasks by first working to solve them individually. They submit their solutions, which are evaluated by the MediaEval organizing committee. Because all researchers working on the same task have used the same data set, the solutions are directly comparable with each other and it is possible to see which approaches provide the best performance for the automatic generation of subject labels. Researchers then gather at a workshop in order to discuss the results, build collaborations and plan approaches for next year. The workshop fosters friendly competition between sites necessary for progress on the issues, but also builds collaboration encouraging sites to bundle their efforts and to avoid duplicating investigation on approaches that have already been shown to be less fruitful.

The MediaEval 2010 workshop was held in Pisa, Italy in October 2010 directly before ACM Multimedia, a large multimedia conference. It was held in a medieval convent "Santa Croce in Fossabanda" that had been converted into a hotel with seminar facilities. A site so evocative of the beauty and the value cultural heritage was particular suited to host researchers focused on the issues that will help improve automatic indexing of tomorrow's cultural heritage content.

Speechless? Not us.

Our proposal for a fourth workshop on Searching Spontaneous Conversational Speech at ACM Multimedia 2010 was accepted today.

ACM Multimedia 2010 Workshop
Searching Spontaneous Conversational Speech (SSCS 2010)
29 October 2010, Firenze Italy

http://www.searchingspeech.org/

The SSCS 2010 workshop is a forum for presentation of recent research results concerning advances and innovation in the area of spoken content retrieval and in the area of multimedia search that makes use of automatic speech recognition technology. Spontaneous, conversational speech occurs in a wide variety of domains and the workshop is relevant for lectures, meetings, interviews, debates, conversational broadcast (e.g., talkshows), podcasts, call center recordings, cultural heritage archives, social video on the Web and spoken natural language queries. The objective of the workshop is to bring together researchers in theareas of speech recognition, audio processing, multimedia analysis and information retrieval for exchange and interaction.