BBC Don't Make Me Evil: We need fragment level access to online spoken audio content

Today, I uploaded something to YouTube I strictly speaking probably should not have. It was a section from a BBC podcast Outriders containing an interview with Heather Marsh about connecting and protecting people by creating social networks from the bottom up rather than from the top down, so that the power and control remains with the individual and not with with an overarching central authority.

In particular, she discusses working with Tribler the open source peer-to-peer client at TU Delft. At 4 minutes and 28 seconds into the interview she states, "It's what we always wanted, it's what the Internet always was supposed to be and we're at the point now where there is no excuse not to have it."

C'mon BBC, there's no way that I am not going to take that sound byte and spread it through my social network. I just don't have that kind of will power. But are you allowing me to do this?

No. The podcast is one monolithic .mp3 file. The times at which the individual interviews (there are three on three different subjects) begin are not given. There is simply no easy fragment level access possible.

And here's where my self control breaks down. I excerpt the interview from the mp3 and upload the thing to YouTube. Twinged by guilt, I generate myself a neat deep link. (As I mentioned in my previous post that touched on deep links as used by YouTube, a deep link is a link that let you jump in to a particular point in an audio file.)

Voila, here is my link for the "there's no excuse not to have Tribler" sound byte:

(except my conscious got the better of me and I deleted it)

Most people I know are so much more likely to click on a deep link then to struggle with the podcast download at:

http://downloads.bbc.co.uk/podcasts/fivelive/pods/pods_20120221-0400a.mp3

Of course with the download it's still Tweetable:

TU Delft P2P client Tribler as heard on BBC Radio 5 "It's what we always wanted" Listen in at 4'28''
http://downloads.bbc.co.uk/podcasts/fivelive/pods/pods_20120221-0400a.mp3

Twitter's link shortening would get that down to 140 characters. But then I have to trust my followers are willing to devote about 10 times more attention to digesting my Tweet, then to other Tweets containing links to streams not downloads.

Not only can I not easily link to my sound byte, I also have no way of finding this reference to Tribler unless I already know this is there, occurring in the middle of an interview. There is no indication of the names of the interviewees and give only very limited information on the topic.

And to aid findability, of course, I turn on the YouTube automatic captioning and see what kind of text transcript the speech recognition will generate.

It's just a click on the little cc icon on the bottom and the transcripts are displayed. You can use the player bar to move quickly through the video and the changing transcript sort of gives you an idea of what the different parts of the interview about. You won't be blown away by the quality of the transcripts, "Tribler" for example is not correctly recognized. Nonetheless you can use them to figure out where to stop, and how to navigate to listen to particular questions.

In short, this workaround instantiates the principle of the intelligent multimedia player. Ask me about the Internet as it always was supposed to be and I would say that there should be no audio content, no interesting sound bytes, buried away without the possibility to find them and to share them easily. We need the fragment level access, the deep links, the player that tells us what is where when we're listening.

Can we get you to do this for us BBC? You might notice that my YouTube solution here is quite ugly and and strictly speaking probably not at all what you have in mind for me to be doing with this content.

But let me assume that the mp3 is online in order to be listened to and shared. Let's continue our search to find new ways to make this possible.

Search your own dogfood

How many hours do I spend writing deliverables and reports? I'd rather not count. Here I am on Friday night with a to do list left over from the week that seems only very vaguely connected to my main mission as a researcher, namely to improve multimedia access systems, especially for spoken audio and video with a speech track.

Sometimes it takes writing a blog entry to refocus on the core values of multimedia search. I was going through the pictures from the Searching Spontaneous Conversation Speech workshop in order to find a good one to add to the latest newsletter report, and dang it, if there weren't so many speaker pictures that we ruined because Florian is crouching in the middle in the front, tending to the laptop that we were using to capture the sound.

At Interspeech we discussed the idea of simply recording all the spoken audio at both the MediaEval 2010 workshop and the SSCS 2010 workshop in order to start an audio corpus of workshops to use for research on meeting retrieval. It sounded like a good idea that we would never have the time to pull off, but sure enough, there we were in Italy, and a network of people came together and brought sound equipment from all over and we had ourselves a system for audio capture. I remember the satisfaction in his voice, when Florian announced "We are now recording six channels". Actually, I remember it because I listened to it on the recording afterwards as we started the laborious process of post-processing and I wondered "Gee, what kinds of things were we talking about next to the main presentations."

So here's the refocus. Florian isn't actually ruining the picture. His presence actually underlines what the speaker is talking about -- the slide reads "The ACLD: Speech-based Just-in-Time Retrieval of Meeting Transcripts, Documents and Websites". We have made such a huge step in this direction that in are own lives we can simply decide to capture our spoken content, everyone at the workshop says, "OK, that's cool" and bang we have more data than we know what to do with.

We also did this at SSCS 2008 in Singapore. The videos were online for a while -- we transcribed them using Nuance Audiomining SDK for speech recognition and made them searable with a Lemur-based earch engine. For awhile, we could visit a website and search our own dogfood, as it were. It seems, however, that the multimedia lifecycle got the better of our content: the system was not maintained and now the videos are no longer available online. I don't know if we'll do much better this year, but the point is that we keep on trying. And we have Florian in the middle of the workshop picture reminding us that this attempt may be time consuming, but it is constitutes the core of our research mission.

Looking for a Scientific Programmer

I'm starting a cool new project on speech-based access to images, but I am in need of a programmer. I'm trying to find the right person -- ideally I would like someone who also had an interest in the process of design and evaluation of the system. The person probably just finished their masters and is trying to get an idea of an area for a PhD, or just generally thinks that one year of experience in the Multimedia Information Retrieval Lab at Delft University of Technology would be enriching. Unfortunately, trying to get someone like this just led to me loosing my dream candidate to a PhD program in Groningen. Yikes! The project's starting on 1 September 2010.

The formal qualifications are listed below. Thanks in advance if you help me find a match between my needs and a candidate.

Profile for scientific programmer in the Delft Multimedia Information Retrieval Lab at the TU-Delft
Contact: Martha Larson m.a.larson@tudelft.nl

• Experience developing web applications using a web development stack, (one of LAMP, Java/Tomcat, ASP.NET/C#)
• Experience with designing HTTP-based server APIs
• Alternatively or in addition: Experience with html/Javascript/AJAX/Flash/Silverlight
• Alternatively or in addition: Interest or experience with Android
• Experience with speech recognition, dialog systems, audio spatialization a benefit
• Experience with programming in a research environment a plus
• Proficiency in English, both spoken and written

The person needs to be an EU citizen, but if I find a great candidate who is not, I am willing to attempt to "battle the system" to get him/her.

P.S. This post represents an experiment in making use of my social network. If was were more up-to-date I suppose I'd be using Linked-In or Facebook, but neither are come naturally to me, somehow.

and I have transcended

Here's a screen shot from my YouTube video about the intelligent multimedia player. I've had YouTube do the automatic transcription, which you can see displayed as a caption at the bottom.

Captioning my video with the YouTube speech recognition service got me thinking about an op-ed piece that appeared in the International Herald Tribute last month: Typing with a Voice by Stewart Wachs. As the title suggests it's about using the computer via a speech recognition interface rather than a keyboard. Unsurprisingly, his speech recognition software generates some wildly off-the-mark transcriptions of what he's saying -- and these serve to make the op-ed quite amusing. But what sticks and doesn't let me go is the final turn of the article, where he talks about a flash of insight that allowed him to start extending empathy to the software. This change of attitude made the mis-recognitions less frustrating allowing him to work with the technology instead of pulling the opposite direction.

He worries that his compassion for the software is misplaced, citing the pathetic fallacy, the mistaken attribution of human characteristics to an inanimate object. The compassion is useful, nonetheless, he observes.

Does this explain my own fascination with speech recognition that has extended over more than a decade now? Is is some sort of innate ability, a gift I have to empathize with the software? Perhaps. If there is indeed compassion involved, more likely it arises not from working with an inanimate software, but rather from the accumulation of interactions with speech recognition researchers I have had over the years. It stems from an appreciation for their creativity and hard work and especially their patience with an external world where word error rate rules and where anything less than 99% accuracy is declared a priori useless. Humans, they patiently point out, don't achieve 99% accuracy in transcription and speech transcripts can be useful for speech retrieval even with error rates over 50%.

Sometimes you administer a bit of tough love, however, like on the day I decided to test drive the new YouTube automatic transcription service. The video I transcribed was recorded using the built-in mic on my Cannon Powershot which was about 2 meters away from me and there was also quite an echo in the kitchen where I recorded the video. If you were squinting at the small screen shot above to read the transcript generated by Google I'll now reveal that it reads, "and I have transcended weeks." Hmm, maybe I have transcended -- reached some sort of a plane where I embrace rather than push away the unexpected outputs of automatic speech recognition. They say, after all, that the speech recognition problem is AI-complete, as difficult as artificial intelligence. Maybe there is no other reasonable choice.

In reality, what I am saying at this point in the video is "...and our friends send us links!" What does this have to do with search? I say "send" YouTube hears "scend" not much of a common ground, but with a syllable level indexing system it would be enough to help locate this quote within the video. That's a lot better than what we have right now.