Affordance

"People," continued the taxi driver driving me to the airport in Dublin, "do the strangest things with chocolate." He paused, reflectively, before adding, "I mean in private."

When I didn't immediately respond, he hurried to explain himself. "You know, a Bounty bar?" I did. "I pick the chocolate off of the outside and then eat the inside separately. Do you do that?" As politely as I could I explained that I didn't like Bounty bars. "What do you do then?" he asked. The best thing that I could come up with was Oreo cookies, that I twist them open and eat out the middle, "A lot of people do that," I added. This puzzled him, until he brightened, "Oh, I heard about this biscuit in Australia and you bite off two of the corners and you drink your tea right through the biscuit. It has some sort of a cream filling that just melts as you drink. It's supposed to be just lovely." He thought for a moment. "It's Tam Tam or Yam Yam or something like that it's called."

I tried to imagine the Tam Tam or the Yam Yam and what it might look like. I was in Perth for about four days after SIGIR 2008, but didn't remember any cookies like that. "Do you suppose," I asked him, "that people just take the biscuit out of the package and look at it and think, 'oh, I should break off two of the corners and drink my tea through it' or was there one person who invented it and then it quickly spread as an idea throughout Australia?"

His response surprised me: he laughed! Then, "It's like the comedian," he pronounced. And then he filled me in: there is a comedian one-liner about watching a chicken lay an egg. "Hey, I think I could eat that!" was the punch line.

And so, I end up discussing with a Dublin taxi driver, the principle of affordance, the ability of an object to be acted on in its environment, and, in a larger definition, communicate its use via its appearance.

In multimedia information retrieval, I am obsessed with affordance in this latter sense. At the first glance or very quickly during interaction, the system should implicitly communicate to the user what it does, what the user can do with what it does and the extent to which it can be trusted to reliably do what it does in all cases.

A few years ago, I believed that a speech retrieval system should not show transcripts to users because users are disturbed by errors. Now we are all a lot further. People are used to reading relatively unedited or unconventional text in text messages, blogs (!) and comments. Now, the level of error can signal to the user that the text has been created by a speech recognizer and how well that speech recognizer can be trusted to capture the spoken content of the audio signal.

But his is negative affordance, the message what can't this system do. It is quite possible that negative affordance is much more challenging to communicate to the user since the space of possible non-uses is not intuitively constrained.

And with biscuits, of course, comes the problem of distributed affordance. What works well once does not continue working well with repeated applications. The package of biscuits should tell you, individually, we are delicious, but if you eat the whole package you won't feel nice and full, but instead you will have an unhappy stomach. Even it was written explicitly on the package, I imagine I would mostly ignore that message.

This is Part III (final part!) of the "Irish Chocolate Discussion", reflections on the conversation I had with a Dublin taxi driver and how that relates to finding things and search systems in general.

Geotags and Geotrails

At the moment I am sitting in a German IC. I'm doing "Duivendrecht" ... "Apeldorn" ... "Hannover" ... "Berlin" and if I had a GPS that was logging my position, you would see that I was doing a certain trajectory at a certain speed. For me there is something soothing about sitting in the train watching Germany roll by. You've made the decision to go there and now all you have to do is wait and it will happen. In this state of doing nothing while doing something, I can unleash my thoughts, do a bit of mental housekeeping and just general feel like I'm absorbed into the German landscape. It's always been like this for me, and I can highly recommend it. If you try it within the Netherlands, you probably have to go to Groningen since the other trajectories would be too short.

In the last entry I mentioned Creative Tourism. This train ride to Berlin is my way of really feeling in touch with Germany -- a way of living and a way of being. If your reading posts in chronological order, you'll recall that our Near2Me concept links up two places people. If enough people take pictures at places with two different geotags, then the place must be related. If you like one, then you like the other. But what I am doing right now is not associated with a single geotag, it is associated with an entire trajectory and also a very specific "train speed". Perhaps our systems will be richer if we include "geotrails", a path with parameters of time and space. These are the doodles we draw and redraw in our travel experiences, patterns that represent something we like to do, but might also represent the type of thing that we should try in the future when we feel inclined to "branch out".

Two other examples for which geotrails could be helpful come to mind. The discussion about chocolate that I had with the taxi driver that drove me to the airport in Dublin on Saturday really stuck with me. But it is one of the things that strikes me nearly every time I am there. You can really talk to the taxi drivers. Not always, but sometimes you have these amazing conversations and that seems to be more important to them than the tip. Once I had a Dublin taxi driver refuse the tip.

I had one conversation where the taxi driver asked me what I did and I said I worked in a multimedia information retrieval lab. "So what do you do there?" I didn't know where to start, so I described to him one of the systems at our lab that processes videos of soccer games and does highlight detection. He thought that was interesting, and I then asked him what he thought counted as a highlight in football. Is it only the goals and the penalties, or what other parts do you want to see if you are watching a summary to the game. He gave a thoughtful answer to this questions. If you would look at my Dublin geotrails, there are a series of characteristic doodles, often ending at the airport that represent these experiences I have had in taxis.

My Amsterdam geotrails, on the other hand, show me giving the taxi stand wide berth. The taxis will drive you in circles there. They have to, sometimes, given the geometry of the city, but they'll add their own embellishments that drive your final price up. The only place that is worse is Brussels.

In Amsterdam, I prefer the bike. You'll see a lot of bike doodles. These are slower than the taxis and also go on roads where cards don't drive. Biking is the authentic Amsterdam experience. Visitors to the city often participate in the local culture by renting bicycles and riding around. This is Creative Tourism the way it was conceived to be carried out.

However, you would see the difference in the geotrails. Amsterdamers bike every day from point A to B and back again. Maybe there is C and D as well, but in general they know their routes, every stop light, every bump, every place where another bike might come out from an unexpected directions. They also know their routes at the time of the day that they characteristically ride them, for example, in the early morning, there was always a characteristic amount of traffic when I biked out from the Lelygracht in the center of Amsterdam where I used to live to the Science Park. Routes are optimized so that they do not take minute longer than they absolutely have to.

Slipping here back into the topic of my last blog, I would like to make the point that the tourist geotrails are totally different, they are slower, different times of the day and involve indirect routes. Tourists wobble a bit, they maybe haven't ridden a bike since they were kids. They stop unexpectedly to consult their maps. They stop for puddles -- and Amsterdamer has seen the route dry the week before and knowing exactly how deep the puddle is going to be rides through it. Tourists also stop for red lights -- Amsterdamers know which lights are conventionally ignored by bicyclists.

Don't recommend to me a quick straight shot to the Science Park if I am on vacation. I want to do the trail with a little wobble that ends me up at the Van Gogh Museum. Is that the authentic Amsterdam experience? Maybe not. But if you push the limit, authenticity is in the eye of the beholder. For the Amsterdamer, the authentic bicycle culture consists of complaining of how badly tourists ride bicycles. Take away commercial tourism and the city loses some of its characteristic spirt, the tension between those that live there and those that play there. I'll leave it to the reader to decide if this would affect Amsterdam's charm.

This is Part II of the "Irish Chocolate Discussion", reflections on the conversation I had with a Dublin taxi driver and how that relates to finding things and search systems in general.

Authentic, Personalized Travel Recommendation

At the moment a lot of my time and attention is being devoted to Near2Me, which is a concept for a travel recommender developed by a colleague of mine, Luz Caballero, for the PetaMedia Network of Excellence. The cool thing about Near2Me is that in not only makes personalized travel recommendations, but it also focuses on authenticity -- the distinctive "spirit of place", as Luz likes to describe it.

The Near2Me concept tackles the problem of long-tail recommendation, it suggests that you go places where relatively few other people have been: Most of the time we refer to it as Off-The-Beaten-Track. The concept links up with the "Creative Tourism" movement, which holds that travel should be participatory and that travelers should hook up with the living culture of the place as manifested in the lives of the people living there.

How do we get at the authenticity of a place? Well, Luz points out that the act of taking a picture is something like adding a tag to a place. The tag is different than other sorts of tags. For example, it is not like the tags we use on Flikcr, since it doesn't have a lot of semantics. Taking a picture is basically a salience tag, it just says, this came to my attention as interesting. Luz did a study of Flickr where she determined that people who travel, even though they of course take pictures of the "must sees", Notre Dame in Paris, for example, also take pictures of interesting things along the way.

Because the motiviation of the people taking these pictures is something more akin to personal documentation than it is to touristic promotion, they provide a valuable source of information about the authentic, the things that we stumble across when we actually are in a place, along the way to reaching another goal.

The next step is simple, you look at the geotags of the pictures (that little piece of metadata produced by a growing number of cameras that records where the picture was taken) and you use the Amazon principle: people who liked X also liked Y. You look for people who have taken pictures where you have taken pictures and see where else they go. Luz' uses the example of markets: if someone liked the "Borough Market" in London, the system would recommend the "Marche Wilson" in Paris. The technique is called "wormholes" and was developed by Maarten Clements.

At this point, you encounter possibly the most challenging part of long-tail recommendation. You want to give people something new and interested, but none of the place recommendations are going to be obvious since you want to stay away from the bestsellers, the commercial destinations. For this reason, its important to be able to explore recommendations further: OK, the system thinks I should like the Nusantara Museum in Delft, but would I really like it? Why? What is it anyway? What can I see and do there?

Two of the ways that users can explore a place are by browsing pictures and browsing people. The Near2Me concept offers a selection of pictures from the recommended place that are carefully chosen to be both diverse and also representative of a place. Also, it offers a selection of people, who by virtue of the pictures that they take and of the popularity of those pictures, emerge as a sort of community expert for a particular topic. How these algorithms work -- and if they are of use in a working prototype is something in the works targeted for publication in future PetaMedia papers.

On Saturday, I flew back from Dublin after having spent a couple days at DCU. On the way to the airport, the taxi driver asked me where I was going.
"Back to Amsterdam." I said. "I came Wednesday and now I am already going back."
"Oh, Amerdam. Did you bring tulips?"
"No, chocolate." I replied, and then added "The funny thing is, is that I'll probably get some chocolate to bring the other direction as well."
"I'll tell you what you got to get." he said. And then launched into description of how amazing the Baileys Irish Cream chocolates are. I've been to Dublin several times, but I still have to ask people to repeat words sometimes because of the Irish English, which he did quite patiently. In the end the picture emerged that the Baileys Irish Cream chocolates are amazing because they are not just chocolatey, but they offer a real Baileys experience as well.

Last time flying back from Dublin I had seen the Baileys Irish Cream chocolates at the airport and had steered clear -- putting Baileys in chocolate was something, I assumed, they had come up with for the tourists, and that I would do well to avoid it. However, the fact that the taxi driver had no particular reason to tell me this and the fact that he was, himself, to me clearly Irish, I shifted my opinion on the Baileys Irish Cream chocolates from "main stream tourist" to "genuine Irish souvenir". Since I was the one that brought up the chocolate, the taxi driver was certainly not doing product placement. Perhaps Baileys is simply very, very smart with this. But if they are, the Baileys Irish Cream chocolates will still remain authentic to me because the whole story happened at a moment where I was devoting a lot of time and attention to Near2Me and thinking a lot about authenticity of place.

This is Part I of the "Irish Chocolate Discussion", reflections on the conversation I had with a Dublin taxi driver and how that relates to finding things and search systems in general.