Criteria for judging a demo in a conference demo session

Mercan Topkara and I are the "Innovation and Demo Chairs" for ICME 2012 to be held 9-13 July 2012 in Melbourne, Australia. We were called upon to organize the decision making process by which the ICME organizers would arrive at the decision of which demo would take home the ICME 2012 best demo award.

The decision is a difficult one because demos in the area of multimedia tend to be radically different in nature. For this reason, I formulated a list of six dimensions to use when judging demos.

1. Clarity: Understandability of the demo paper and the presentation.
2. Realization: Well implemented, robust, good use of technology.
3. Innovation: Addresses a problem that has not yet been tackled (or has proven difficult to solve).
4. Impact: The number of people the technology potentially touches and the importance of its influence on their lives.
5. Representativity: Centrality to the topics covered by the conference (in this case ICME)
6. Magic: How closely the technology fills the description, "Any sufficiently advanced technology is indistinguishable from magic" (cf. http://en.wikipedia.org/wiki/Clarke%27s_three_laws)

Number 6 is basically a wild card that makes it possible to introduce in a controlled way that factor of je ne sais quoi, which seems to slip into the considerations made when judging demos in any case.

In practice, another factor that always seems to be important is how close the demo is to a working system that is or is about to be deployed in the real world. Also, when judging demos it seems that one is always trying to project forward: how important will this technology be five or ten years from now? Will the passage of time reveal that it is a disruptive technology? (Or, as Wikipedia prefers to call it disruptive innovation?)

For a list of the demos to be presented at ICME 2012, see the ICME 2012 Demo page.

I am dating the post 1 June, when I formulated the list of criteria. It's later now, but time seems to have simply gotten away from me, not surprising given the ICME 2012 Time Machine Session.