Saturday, April 23, 2011

Proto-semantics: Where cognition meets language

A recent Nature paper entitled Evolved structure of language shows lineage-specific trends in word-order universals has made quite a splash in the media and has some important implications for the study of linguistics and application of linguistic principles. The splashy aspect of the paper is that it allows the media to declare the discovery of purported fundamental flaws in the theory of Universal Grammar.

The more subtle implications of the research are highlighted in an Wired Science article entitled Evolution of Language Takes Unexpected Turn. This article quotes Michael Dunn, author of the Nature paper, as stating, “What languages have in common is to be found at a much deeper level. They must emerge from more-general cognitive capacities.”

This finding suggests that if we indeed are interested in understanding our human language capacity, it is time to return to again give serious consideration to proto-semantics, the pre-lexical dimensions of meaning arising from the interaction of our human perception and the real world in which we live. The commonality underlying the way we speak arises from the similarity of our cognitive processes, which in turn can be traced to the structure and function of our brains. The way in which proto-semantics is encoded into the wide range of human languages and the variety of the evidence that we can observe of its existence are re-emerging as the more critical research questions. Whether we need a universally applicable syntactic mechanism to explain language may at this point be less relevant.

As a scientist, I personally renegotiated my relationship with Universal Grammar 15 years ago while doing the field work in theoretical linguistics on Baule in Cote d'Ivoire. No amount of pushing and shoving would fit the verb series constructions that I was studying into a framework determined exclusively by settings of universal language parameters previously identified in other languages. It appeared to be a classic case of West Africa Wins Again. My laptop held up wonderfully under humidity, sand and travel in the Cote d'Ivoire -- but my fine theory failed miserably.

One aspect of Baule that I looked at in detail is the conditions under which it is possible to omit a pronoun object in a sentence. In answer to the question "Did you beat the drum?" one replies "Yes, I beat", which must be translated into English as "Yes, I beat it". You leave out the pronoun because (simplifying somewhat) the drum is an object whose purpose is to be beaten (in the sense of played). The functional relationship in the real world as perceived by humans makes the verb and its object inextricable with respect to their meaning. The pronoun is considered redundant and can (in fact, must be since the effect is encoded into the grammar of the language) left out.

Contrast that with "Did you see the drum?" and the answer "Yes, I saw it". Here, the acting of seeing and the drum itself do not have a tight connection in the cognitive conceptualization of the world. The drum is seen because it has a physical existence, but it is not explicitly designed to be seen nor can it be considered an object typical of the sorts of objects that are seen by humans. In the answer "Yes, I saw it." it is not possible to omit the pronoun.

The syntax of Baule failed to fit into my neat universal framework, but I learned an extraordinary amount about the range of phenomena that "register on the radar" of human cognition and naturally also find their way into language. Other languages express these phenomena in other ways, or chose not to express them at all. However, that proto-semantic generalizations can be found to exist at all and which generalizations exist is both interesting and useful.

What does this have to do with search (the topic of this blog)? If proto-semantics is a reflex of human perception of the real world, we should expect that its influence impacts not only relatively low level syntactic processes, but also the more complex ways in which humans communicate and exchange information: information creation, curation and retrieval. If we can push forward this line of inquiry, a full answer to this question will surely be important enough to make its own splash.

Sunday, April 17, 2011

From the Rialto: On the Relevance of Geotags

This weekend I am in Venice in between an Networked Media and Search Systems Concertation meeting and ICMR 2011. I am in the processing of preparing the slides for a talk [1] that I will give on Tuesday morning on tagging and geotagging. Once again, I find myself busy with the question of what does it mean for a geotag to be relevant to a photo.

At first blush, the answer to this question is obvious: A geotag is relevant to a photo if it gives the exact longitude and latitude of the position of the camera when the photo was taken. I have this information for the photo above, which was taken earlier today -- but it reflects so little of the larger story. On the one hand our cameras (in this case we're using a new Android phone) pin us down and on the other they tell not much at all.

I originally asked this question about the relevance of geotags after the talk of Pavel Serdyukov at SIGIR 2009, "Placing flickr photos on a map." I garbled the formulation, Pavel didn't understand it, and I have since come to understand the large number of dimensions involved in the answer. "Where does a photo truly belong on a map?" isn't an obvious question and doesn't have an obvious answer, either.

It's generally accepted that knowing the position of the photographer underspecifies the location of the photo since it is not known in which direction the camera was pointing or on what it was focused. It could possibly be the case that the subject of the picture is quite far away from the photographer, making the geo-coordinates of the photographer not the most relevant position to the photo in terms of the depicted content.

The photo from today is a good example. It was taken in Venice, but it is also relevant to Trento, since I am here in Italy on a trip to Trento and only happened to come to Venice for the weekend. I will remember the trip as a "Trento" trip. But the Trento association goes beyond my own personal sense of the proper place of the photo -- anyone who was on a business trip to Trento might find Venice related since they are possibly flying into a Venice airport or like myself also planning a touristic stay in Venice.

The sign in the background says "Per Rialto" and is pointing the way towards the Rialto bridge. We're on our way to the bridge and have taken the photo along the way -- it is relevant to the geocoordinates of the Rialto bridge in the sense that this was the final destination of the walk.

The photo was taken for my family back home. My great uncle is sick and in the hospital at this moment. He is a central node in my extended family and ties us together across space and also across time -- and for years has passed along to us the lore of the generations that preceded us -- a mixture of stories, poetry and family values -- a nourishing flow that sustains us and binds us together. One of the things that he does is write email newsletters to the family members and this newsletter has a section that is called "From the Rialto", which describes the current activities of various family members. Here I am sending my own thoughts out from the Rialto to my family back home and my great uncle in the hospital. The picture has more to do with them and where they are than here.

Further, I don't really remember why the section of the newsletter is called "From the Rialto". It might not be the Rialto in Venice (which is actually an entire district and not just a bridge). It might be an entirely different Rialto. My picture is actually related to the original Rialto used by my uncle, except that I don't know where that is. I will admit that there are probable not a lot of photos out there that enjoy this sort of relationship to place -- but the main point is the existence of possibly relevant places that the person who takes the photo is not necessarily aware.

Actually, I just wanted a picture with a Rialto sign in the background. These signs are numerous in Venice...in fact they sell T-shirts on which they are depicted. In a certain way, the photo is a "Woman in from of Rialto sign" picture and is also relevant to anywhere else a Rialto sign is located. In a way, I am standing at place that has many different geo-coordinates.

The renown of the Rialto bridge is attested by the numerous replicas that exist, for example, in Las Vegas. The original and its imitators are related -- photos of one are of interest to people visiting the location of the other. It would be useful for a picture of the Las Vegas Rialto bridge to bear a tag that relates it to the one here in Venice.

I briefly entertained the idea that it is generally accepted that the relevant geocoordinates for a photo are the ones at which the photographer is standing, since those geocoordinates would be considered relevant to the widest range of people, i.e., be the most objective. Indeed, this is the relevance criteria for geotags that we apply within the MediaEval Placing Task. My association of the photo with the folks back home is relevant to a handful of souls, whereas anyone could recognize the connection with this corner of Venice. However, replicas are part of a common shared culture across the world and the connection between an image of a replica and the geo-coordinates of the original could be considered of possible interest to anyone and not just a personal circle.

Finally, there is the question of the relevant geographical resolution. For me, this is a "Venice" picture. I don't need to know the exact corner again -- nor will I try to find it. We overheard a group of Americans talking about their vacation -- and noticed that they referred to themselves not as being in Venice or in Italy, but in Europe. Europeans hardly ever reflect upon the fact that they are in Europe...they take it for granted.

Wandering through Venice its tempting to just fall into people watching -- making up small stories about couples out enjoying the good weather and the beautiful city: how they met, how they fell in love and what the future has in store for their mutual happiness. The relevance of the geo-coordinates of the photographer is obvious for those who want to take people watching to the next level -- if the couple standing next to us a bridge near San Marco uploads their pictures to Flickr, we will know even then more about them...we could check if their trip to Venice provided the solid basis for a stable relationship, or whether they ended up parting ways. Taking the people watching to the extreme: It would finally be possible to check whether kissing under the Bridge of Sighs does its work in guaranteeing everlasting bliss.

[1] Larson, M., Soleymani, M., Serdyukov, P., Rudinac, S., Wartena, C., Murdock, V., Friedland, G., Ordelman, R. and Jones, G.J.F. Automatic Tagging and Geotagging in Video Collections and Communities. ACM International Conference on Multimedia Retrieval 2011.