Advances in Multimedia Retrieval Tutorial at ACM Multimedia 2010

I worked a long day today and now I'm home and have eaten dinner and thinking about how to relax. I'd like to watch an episode of Merlin on the Internet. Preferably legal and one I've never seen before -- wouldn't mind paying if it was a site I trusted. That seems like a pretty complicated search and my prediction is that it will lead to frustration. So here I am writing in my blog instead.

Today the longplay description of our upcoming tutorial on Frontiers in Multimedia Search went online. We want to start out by addressing the question of how can multimedia search benefit people's daily lives, at work and otherwise. I'm feeling rather a strong need for the benefit of multimedia at the moment. If I can't have my Merlin, right now I wouldn't mind browsing back through recordings of the SIGIR presentations that I heard last week -- and maybe some of the ones that I have missed.

Then we are planning to go on to take a look at new approaches to multimedia retrieval that we divide into three categories (I include a couple of my own notes on each):

• Making the most of the user: In motion on the Internet we dribble information behind us. We tag, we query, we click, we brush over a page without a second glance. We have the capacity to glance at a set of snippets, glaze over what does not interest us to find what does. Making the most of the user is about letting the search engine turn the computational crank and do the look ups, leaving those fine-grained semantic judgments to the human brain.
  
• Making the most of the collection: Sometimes the collection can speak for itself. Pseudo-relevance feedback may dilute our queries, but it also is a valuable tool for increasing recall. And then there is collaborative filtering: Making use of patterns that we as users leave behind -- but now at the collection or community level.
  
• Making the most of individual items: What is important here is how you can do the best that you can with noisy sources of features (speech recognition, visual concept detection) to represent items. You don't need to necessarily provide a complete representation of an item -- information that helps distinguish items or keeps them from getting confused can sometimes be a big help.

All in all, the aim is to present our favorite new multimedia search work -- working to inject new techniques and perspectives from the IR community and speech community into multimedia. And, of course, develop our own understanding of multimedia search along the way. Maybe I will then also feel better equipped to find something that I could watch now that would make me as happy as Merlin. Or at least to understand exactly what is still missing...

Relevant to the query "List of Internet Video Genres"

From the perspective of automatic multimedia content analysis, there is vast difference in visual content between video that was produced for the purpose of being understood and absorbed by an audience and video that was captured with no explicit communicative intent. If two people shake hands in a film, the audience will know that they are shaking hands and it will fit with the dialogue in the sound track and with the over all story. If two people shake hands on a surveillance video one can occlude the other or perhaps they're passing a cigarette lighter who knows.

It seems like creator intent is an important clue for visual indexing of video for retrieval. But the example above is simplistic. If we want to find video on the internet, there are a whole range of intents between film and surveilance. What are these varieties? Today I thought I could type "list of internet video genres" into my favorite mainstream search engine and have it spit me out a list of things that we as Internet users do when we make video. That didn't happen. I spend some time in amused pondering over the juxtaposition in The six most baffling genres on YouTube. But I wanted something a bit more comprehensive than that (with a different tone), so I'm posting my own list.

Captured video: Walking through the room and not knowing the camera is on. Interesting as a curiosity, but serves no specific purpose.
Life-log: I know the camera is on but don’t really think about it. Serves the purposes of off-line memory.


Surveillance video: I put the camera in some particular place to capture the scene, but the people in the scene don’t pay attention to it. Serves the purpose or providing extra ears and eyes.


Home video: I am obviously holding a camera and pointing at people. Not trying to do anything but “get the feel of the situation on video” Serves the purposes of off-line memory. The act of making the video is also inherently entertaining and it might not necessarily be watched.


Event: I am documenting an event, like a wedding. The video is meant to portray both the compliance of the event to social convention and also its uniqueness. Ideally, I want to see the face of the bride and groom and hear the “I do”. Serves the purpose of memory, but may also be considered a public document that attests to status.


Meeting video/lecture video: I know the camera is on but don’t really think about it. Serves the purposes of off-line memory and possibly institutional record. Used to rewatch things that might have been missed the first time. The camera has a specific position – other material such as slides or white board shots might be present.


Testimony: A narrator recounts and experience. Spoken audio is unscripted but declarative factual statements. The narrative is usually temporally organized. The visuals are “convenience visuals”, but there are typical camera angles: frontal shot, shot of interviewer in dialogue with the interviewee. Viewer acquires some declarative knowledge, but basically, it’s an impression of the situation. 


How-to video: Demonstrates how to do something. The visuals are key, with the camera angle chosen to give maximal information. Items depicted in the visual tract have a high probability of being named in the speech track. After watching this video the viewer is intended to have procedural knowledge of the task.


Learning video: The video acts out scenes and viewers are invited to put themselves into the scene. This video is a surrogate for experience. Here, the camera angle is carefully chosen and any spoken audio must be clearly captured (esp. in the case of language learning videos.)
 Again, viewer acquires procedural knowledge, but it is via vicarious doing and not via showing. In contrast to how-to video, learning videos are only intended to be watched once.

Interview/review: I want to get someone’s view or opinion or tell my own. Can be planned or relatively unscripted. Contain opinions or attitudes. Factual statements are made to support opinions. The visuals may be convenience visuals, but may be planned convey the feel of that person.


Report: Following a script, I report a certain even that happened. The statements I make are factual. The visuals provide depictions of the objects mentioned (broadcast news). In the end the viewer has acquired declarative knowledge.


Documentary: Reports a sequence of events subordinate to some sort of ordering. Temporal ordering is common, or they may be ordered in order to support a thesis. Documentaries include a narrative line: they open questions and resolve them. In the end the viewer has acquired declarative knowledge.


Film (or TV Series): Narrative created for the purposes of entertainment, but may have other elements (didactic, community memory). The basis of film is a complex “contract” between the filmmaker and the viewer, which rests on a series of established conventions that have been developed over the history of filmmaking (the literature traces this system of conventions back into novels). The nature of this contract varies from film genre to film genre. Narratives are created by setting up viewer expectations and then either fulfilling the expectations or failing to fulfill them. Scenes are carefully composed to carry out scene setting, introduce characters and to depict events. Organization can be temporal or otherwise. Shots are set up so that the viewers understand what is going on (in all but a few exceptions, the main action in the shot will be readily visible, for example, the moment that he passes her the gun, the moment that she kisses him). In addition to understanding the plot, the film aims to create a mood for the viewer. Conventions are used to create mood include music, timing shifts (quick shot sequence used to portray the passage or time), lighting, camera angles. Further, films are created to delight the viewers with their film craft, which can involve strict adherence to filmmaking principles (including references to other films) or creative breaks with convention.


Art: With art the contract between the viewer and the creator is not as complex as with film. In fact, it can be considered utterly simple. The viewer simply has to agree, “this is art”. The impression the video has on the viewer is decoupled from the intention of the creator to a greater extent than in film. Art closes the circle and resembles captured video in that it the video is an object in of itself. It assumes a purpose in the act of viewing. Art events are not necessarily depicted so that the viewer understands the “plot”, the conveyance of a certain mood may be highly viewer dependent and there is no narrative. If there is a speech track, there is no predictable coupling between the speech track and the visual channel.



Object: Sometimes we make video and we don't know why. Neither videographer or viewer would readily commit to the "art" label. Even a video that we have made ourselves become objects that inhabits the world of objects, things we come across, think about, try to fit in to the larger pictures. Some of these videos are the ones whose existence in an of itself explains and expands the role of video. Here perhaps there is only a person and a camera and it is not appropriate to speak of intent. Video happens.

In sum, the intent of the creator can be used as a basis of a typology containing many different kinds of video. Each is different from the other in several important respects, including, (1) what information is packaged by the creator into the visual channel and (2) what the relationship is between the visual channel and the audio track. In light of this typology, it is rather curious that we consider multimedia information retrieval to be a single discipline. Instead, every genre presents us with a unique set of challenges -- an entirely different range of issues that need to be face to provide retrieval algorithms that succeed in meeting users' needs.

Getting the words right

The final day of Interspeech 2009 here in Brighton. It's been a great conference and each and every keynote has been well worth getting up for. This morning, Mari Ostendorf talked about "Transcribing Speech for Spoken Language Processing." Interspeech encompasses a staggeringly broad spectrum of perspectives on speech research and technology. For every point here, there is an immediate counterpoint, and it was without doubt under influence of this chorus that the opening slide of the keynote this morning displayed a long-play version of the title reminding the audience that they would be hearing about transcribing human-directed human speech, as opposed to speech that humans produce to communicate with computers.

The message from the keynote that will ring longest in my ears was, "The goal of speech transcription is information access." This leaves open of the course, the question of what is the information and what is the access when it comes to content that contains the spoken word. I find myself compiling little lists of domains in which information encoded in spoken audio could be important: podcasts, video diaries, lifelogs, meetings, call center recordings, social video networks, Web TV, conversational broadcast, lectures, discussions, debates, interviews and cultural heritage archives, home videos, photo annotations, video conferences. These lists invariable end with etc. etc. etc. And what constitutes access (keyword search, retrieval, question answering, browsing, recommendation...) is another question to which we can't give a closed-set answer.

My personal experience doesn't really support the idea that we need to push the envelope. The last video I watched I found because a link was sent to me by my cousin. The content of the video was a short clip of her new cat purring. No real access problem there. No information either. The purr did not inform me in the conventional sense. In fact, there wasn't much human speech involved at all. Nonetheless, I found the content supremely worthwhile of my watching time. Although my own multimedia access needs are a string of examples of this pre-solved sort, I do agree that the challenge of access to speech-based information is a serious one and will require a great deal of effort to address.

The full phrase in Ostendorf's slide read, "The goal of speech transcription is information access, not just getting the words right." But maybe it is about "getting the words right". The words referred to are, presumably, white-space delineated grapheme strings, lexical words, citation forms. But we can also see a word as the totality of knowledge that a human needs to possess in order to deploy it in human-to-human communication. There may be a limit to how far we can go beyond that sort of word and still remain within what is meaningful in the context of our information access needs.

We can go for prosody, for speech act, subjectivity, affect, but in the end we'll never capture the "you had to have been there" component of understanding. And already the moment of that particular purr video has passed and my next need for video content will be for a new one.

The Netbook Effect

"Give a laptop. Change the world." are the welcoming words of the website of One Laptop Per Child. A Wired article entitled "The Netbook Effect: How Cheap Little Laptops Hit the Big Time" (Wired, March 2009) tells the story of how the low cost no-frills netbook evolved from the One Laptop Per Child initiative and predicts that netbooks will constitute 12% of the laptop market in 2010.
It sort of takes my breath away that I might actually live to see One Laptop Per Child happen. It gives me hope that we might eventually move not only as individuals, communities and countries, but as an entire species beyond subsistence level concerns.
At the same time, another part of my brain is formulating questions about what happens as an increasing proportion of personal machines are "thin" in the sense that they have low processing power and low storage capacity. The vision of democratizing the supply of services with high computational loads by distributing them over private citizens with unused processing capacity may have to be abandoned. Do we want to encourage a future in which we must rely on a distant center to store and swap our video content? Do we want to close the door on the possibility of internet content search that is supported by a million modest contributions of storage space and cpu cycles? Are we ready to give up our private capacities and allow resource rich hands to further accumulate a power monopoly removed beyond influence of the individual?

WIAMIS 2009 in London

Attended the International Workshop on Image Analysis for Multimedia Interactive Services a.k.a. WIAMIS 2009. The PetaMedia task force organized a special session to showcase topics related to combining multimedia content analysis with user contributed information and social network structure. We presented some initial work on combining speech-based indexing features with low level visual information for improved video retrieval.