Big Data as Fast Data

Last Thursday I was at a "sounding board" meeting of the Big Data Commission of the Royal Netherlands Academy of Arts and Sciences. This post highlights some points that I have continued to reflect upon since the meeting. 

According to Wikipedia, "Big Data" are data sets that are too large and too complex for traditional data processing systems to work with them. Interestingly, the people who characterize "Big Data" in terms of volume, variety and velocity, often underemphasize, as the Wikipedia definition does, the aspect of  velocity. Here, I argue it is important not to forget that Big Data is also Fast Data.

Fast Streams and Big Challenges
Because I work in the area of recommender systems, I quite naturally conceptualize problems in terms of a data stream rather than a data set. The task a stream-based recommender system addresses is the following: there is a stream of incoming events and the goal is to make predictions on the future of the stream. There are two issues that differentiate stream-based views of data from set-based views.

First: the temporal ordering in the stream means that ordinary cross-validation cannot be applied. A form of A/B testing must be used in order to evaluate the quality of predictions. Online A/B testing has implications for the replicability of experiments.

Second: at any given moment, you are making two intertwining predictions. One is the prediction of the future of the stream. The other is how much, if any, of the past is actually relevant in predicting the future. There are two reasons why the information in the past stream may not be relevant to the future: external and internal factors.

External factors are challenging because you may not know they are happening. A colleague doing medical research recently told me that when deductibles go up people delay going to the doctor, and suddenly the patients that are visiting the doctor have different problems, simply because they delayed their visit. Confounding variables of course exist for set-based data. However, if you are testing stream-based prediction online, you can't simply turn back the clock and start investigating confounding variables: it's already water under the bridge. As much as you may be recording, you cannot reply all of reality as it happened.

Internal factors are even tougher. Events occurring in the data stream influence the stream itself. A common example is the process by which a video goes viral on the Web. In this case, we have a stream of events consisting of viewers watching the video. Because people like to watch videos that are popular (or are simply curious about what everyone else is watching) events in the past actually serve to create the future, yielding an exponentially growing number of views. These factors can be understood as feedback loops. Another important issue, which occurs in recommender systems, is that the future of the stream is influenced by the predictions that you make. In a recommender system, these predictions are shown to users in the form of recommended items, and the users create new events by interacting with these items. The medical researcher is stuck with this effect: she cannot decide not to cure patients, just because it will create a sudden shift in the statistical properties of her data stream.

Time to Learn to do it Do it Right
In short, you are trying to predict and also to predict whether you can predict. We still call it "Big Data", but clearly we are at a place where the assumption that data greed pays off ("There's no data like more data") breaks down. Instead, we start to consider the price of  Big Data failure ("The bigger they are, the harder they fall").

In a recent article Trump's Win Isn't the Death of Data---It Was Flawed All Along, Wired concluded that "...the data used to predict the outcome of one of the most important events in recent history was flawed." But if you think about it: of all the preposterous statements made during the campaign, no one proposed that the actual election be cancelled since Big Data could predict its outcome. There are purposes that Big Data can fulfill, and purposes for which it is not appropriate.

The Law of Large Numbers forms the basis for reliably repeatable predictions. For this reason, it is clear that Big Data is not dead. The situation is perhaps exactly the opposite: Big Data has just been born. We have reasons to believe in its enormous usefulness, but ultimately its usefulness will depend on the availability of people with the background to support it.

There is a division between people with a classic statistics and machine learning background who know how to predict (who may even have the technical expertise to do it at large scale) and people who, on top of a classical background, have the skills to approach the question of when does it even make sense to be predicting. Only the latter are qualified to pursue big data.

The difference is perhaps a bit like the difference between snorkeling and scuba diving. Both are forms of underwater exploration, and many people don't necessarily realize that there is a difference. However, if you can snorkel, you are still a long way from being able to scuba dive. For scuba diving, you need additional training, and more equipment, and a firm grasp of principles that are not necessarily intuitive, such as the physiological effects of depth and the wisdom of redundancy. There is a lot to be achieved on a scuba dive, that can't be accomplished by mere snorkeling: but the diver needs resources to invest, and above all needs to have the time to learn to do it right.

No Fast Track to Big Data
These considerations lead to the realization that although Big Data streams may be in and of themselves incredibly quickly changing, the overall process of making Big Data useful is, in fact, very slow. Working in Big Data requires an enormous amount of training going beyond a traditional data processing background.

Gaining the expertise needed for Big Data also requires understanding of domains that lie outside of traditional math and computer science fields. All work in Big Data areas must start from a solid  ethical and legal foundation. Civil engineers are in some cases able to lift a building to add a foundation.  With Big Data, this possibility is excluded.

To illustrate this point, it is worth returning to consider the idea of replacing the election with a group of data scientists carrying out Big Data calculations. It is perhaps an extreme example, but it is one that makes clear that ethical and legal considerations must come before Big Data. The election must remain the election because on its own a Big Data calculation has no way of achieving the necessary social trust necessary to ensure continuity of the government. For this we need a cohesive society and we need the law. Unless Big Data starts from ethics and from legal considerations, we risk time and effort developing a large number of algorithms that are solving the wrong problems.

Training data scientists while ignoring the ethical and legal implications of Big Data is a short cut that is tempting in the short run, but can do nothing but harm us in the long run.

Big Data as Slow Science
The amount of time and effort needed to make Big Data work, might lead us to expect that Big Data should yield some sort of Big Bang, a real scientific revolution. In fact, however, it are the principles the same old scientific method of centuries that we return to in order to define Big Data experiments. In short, Big Data is a natural development of existing practices. Some have even argued that data-driven science pre-dated the digital age, e.g., this essay entitled Is the Fourth Paradigm Really New?

However, it would also be wrong to characterize Big Data as business as usual. A more apt characterization is as follows: Before the Big Data age scientific research proceeded along a the conventional path: researchers would formulate their hypothesis, design their experiment, and then as the final step collect the data. Now, the path starts with the data, which inspires and informs the hypothesis. The experimental design must compensate for the fact that the data was "found" rather than collected.

Given this state of affairs, it is easy to slip into the impression that Big Data is "fast" in the sense that it speeds up the process of scientific discovery. After all, the data collection process, which in the past could take years, can be carried out quickly. If the workflow is implemented, a new hypothesis could be investigated in a matter of hours. However, it important to consider how the speed of the experiment itself influences the way in which we formulate hypotheses. Because there is little cost to running an experiment, there is little incentive to put a great deal of careful thought and consideration into which hypotheses we are testing.

A good hypotheses is one that is motivated by a sense of scientific curiosity, and/or societal need and that has been informed by ample amounts of real-world experience. If there is negligible additional cost to running an additional experiment, we need to find our motivation for formulating good hypotheses elsewhere. The price of thoughtlessly investigating hypotheses merely because they can be formulated given a specific data collection is high. Quick and dirty experiments lead to mistaking spurious correlations for effects, and yield insights that fall short of generalizing to meaning real-world phenomena, let alone use cases.

In sum, we should remember the "V" of velocity. Big Data is not just data sets, its also data streams, which makes Big Data also Fast Data. Taking a look at data streams makes it easier to see the ways in which Big Data can go wrong, and why it requires special training, tools, and techniques.

Volume, variety, and velocity have been extended by some to include other "Vs" such as Veracity and Value. Here, I would like to propose "Vigilance". For Big Data to be successful we need to slow down: train people with a broad range of expertise, connect people to work in multi-skilled teams, and give them the time and the resources needed in order to do Big Data right. In the end, the value of Big Data is the new insights that it reveals, and not the speed at which it reveals them.

CrowdRec 2015 Workshop Panel Discussion: Crowdsourcing in Recommender Systems

The CrowdRec 2015 Workshop on Crowdsourcing and Human Computation for Recommender Systems was held this past Saturday at ACM RecSys 2015 in Vienna, Austria. The workshop ended with a fish bowl panel on the topic of the biggest challenge facing the successful use of crowdsourcing for recommender systems. I asked the panelist to take a position as to the nature of this challenge, was it related to algorithms, engineering or ethics. During the panel the audience took crowdsourced notes about the panel on titanpad.

After the workshop I received two comments that particularly stuck in my mind. One was that I should have told people that if they contributed to the titanpad notes, I would write a blogpost summarizing them. I was happy that at least someone thought that a blog post would be a good idea. (I hadn't considered that having ones thoughts summarized in my blog would be a motivational factor.) The other comment was that the panel question was not controversial enough to spark a good discussion.

In response to these two comments here now a summary/interpretation of what was recorded in the crowdsourced notes about the panel.

• The biggest challenge of crowdsourcing is to design a product in which crowdsourcing adds value for the user. Crowdsourcing should not be pursued unless it makes a clear contribution.

• The way that the crowd uses a crowdsourcing platform, or a system that integrates crowdsourcing is essential. Here, engagement of the crowd is key, so that they are "in tune" with the goals of the platform, and make a productive contribution.

• The biggest challenge is the principle of KYC. Here, instead of Know Your Client, this is Know Your Crowd. There are many individual and cultural differences between crowdmembers that need to be taken into account.

• The problem facing many systems is not the large amount of data, but that they data is unpredictably structured and in homogenous, making it difficult to ask the crowd to actually do something with it.

• With human contributors in the crowd, people become afraid of collusion attacks that go against the original, or presumed. intent of a platform. A huge space for discussion (which was not pursued during the panel) opens about who has the right to decide what the "right" and "wrong" way to use a platform.

• Crowdwork can be considered people paying with their time: We need to carefully think about what they receive in return.

• With the exception of this last comment, it seemed that most people on the panel found it difficult to say something meaningful about ethics in the short time that was available for the discussion.

In general, we noticed that there are still multiple definitions of crowdsourcing at play in the community. In the introduction to the workshop, I pointed out that we are most interested in definitions of crowdsourcing where crowdwork occurs in response to a task that was explicitly formulated. In other words, collecting data that was create for another purpose rather than in response to a taskasker is not crowdsourcing in the sense of CrowdRec. It's not uninteresting to consider recommender systems that leverage user comments collected from the web. However, we feel that such systems fall under the heading of "social" rather than "crowd", and reserve a special space for "crowd" recommender systems, which involve active elicitation of information. It seems that it is difficult to be productively controversial, if we need to delineate the topic at the start of every conversation.

At this point, it seems that we are seeing more recommender systems that involve taggers and curators. Amazon Mechanical Turk, of course, came into being as an in-house system to improve product recommendations, cf. Wikipedia. However, it seems that recommender systems that actively leverage the input of the crowd still need to come into their own.

See also:

Martha Larson, Domonkos Tikk, and Roberto Turrin. 2015. Overview of ACM RecSys CrowdRec 2015 Workshop: Crowdsourcing and Human Computation for Recommender Systems. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys '15). ACM, New York, NY, USA, 341-342.

Choosing movies to watch on an airplane? Compensate for context

For some people, an airplane is the perfect place to catch up on their movies-to-watch list. For these people there is no difference between sitting on a plane and sitting on the couch in their living room.

If you are one of these people, you are lucky.

If not, then you make want to take a few moments to think about what kind of a movie you should be watching on an airplane.

These are our two main insights on how you should make this choice:

• Watch a movie that uses a lot of closeups (or relatively little visual detail), is well lit, and moves relatively slowly so that you can enjoy it on a small screen.
• Watch something that is going to engage you. Remember that the environment and the disruptions on an airplane might affect your ability to focus, and, in this way disrupt your ability to experience an empathetic relationship with the characters. In other words, unless the plot and characters really draw you in movie might not "work" in the way it is intended.

For an accessible introduction to how movies manipulate your brain see the Wired series on Cinema Science article:

http://www.wired.com/2014/08/how-movies-manipulate-your-brain
At the perceptual level your brain needs to be able exercise its ability to "stitch things together to make sense". It's plausible that this "stitching" also has to be able to take place at an emotional level. Certain kinds of distractors can be expected to simply get in the way of that happening as effectively as it is meant to.

When we began to study what kinds of movies that people watch on planes we used these two insights as a point of departure. We started with these insights after having made some informal observations about the nature of distractors on an airplane, which are illustrated by this video.

At the end of the video, we formulate the following initial list of distractors, which impact what you might want to watch on an airplane.

1. Engine noise
2. Announcements
3. Turbulence
4. Small screen 
5. Glare on screen
6. Inflight service
7. Fellow travelers
8. Kid next to you

In short, when choosing a movies to view on the airplane, you should pick a movie that can "compensate for context", meaning that you can enjoy it despite the distractors inherent in the situation aboard an airplane.

We are looking to expand this list of distractors as our research moves forward.

Our ultimate goal (still a long way off) is to build a recommender system that can automatically "watch" movies for you ahead of time. The system would be able to suggest movies that you would like, but above and beyond that the suggested movies would be prescreened to be suitable for watching on an airplane. Such a system would help you to quickly decide what to turn on at 30,000 feet, without worrying that half way through you will realize that it might not have been a good choice.

Since we started this research, I have been paying more an more attention to the experiences that I have with movies on a plane. Here are two.

• On a recent domestic flight: The woman next to me started to watch Penny Dreadful. She turned it off about ten minutes in. I then also tried to watch it. I really like the show, but it's meant to be dark, gory, and mysterious. These three qualities turned into poorly visible, disconcerting, and confusing at 10,000 feet. This is what I am trying to capture in the video above.
• On a recent Transatlantic flight: The man sitting next to me turned on his monitor, and turned on The Color Purple as if it were on his watch list. It's probably on most people's movies-to-watch list, so this seems like a safe choice. However, I was trying to review a paper, and was subject to over two hours of unavoidable glimpses of violence on a screen a few feet from my own. It's the emotional impact of these scenes that make it a great movie. By the same token, you might not want to be watching it on a plane, especially if you are not going to experience the entire emotional arc. (The movie should be watched with full focus, from beginning to end.)

Until now, the work on context-aware movie recommender systems that I have encountered has recommended movies for situations that are part of what is considered to be "normal" daily life, e.g.,  watching movies during the week vs. on the weekend, watching movies with your kids vs. with your spouse. We need more recommender system work that will allow us to get to movies that are suitable for less ideal, more unpleasant, perhaps less frequent situations.  Why waste a good movie by watching it in the wrong context? And why suffer anymore than necessary while on an airplane?

The work is being carried out within the context of the MediaEval Benchmarking Initiative for Multimedia Evaluation, see Context of Experience 2015. It owes a lot to the CrowdRec project, which pushes us to understand how we can make recommenders better by asking people explicitly to contribute their input.

Drink recommendation

Within the last ten days I've been in Asia, Europe and North America. I've taken jetlag to a new level. Usually there is a reference point, you can say, "It's past midnight in the Netherlands at the moment, my internal clock thinks it's past my bedtime and that's why I am so tired." Now I have no clue why time my internal clock reads.

At the grocery store, I just picked out a four pack of energy drink in order to try to jump start myself and get re-aligned with the cycle of the sun at my current location. I stood for ten minutes in front of the selections, looking at the cans and then reading the labels. I wanted something not too expensive, sugar free and also with guarana. A Brazilian colleague had recommended guarana as one of the best "pick up" ingredients you can get in an energy drink.

What I could use is a good drink recommendation system. The Asian part of this odyssey took place in Tokyo, and the following video was what YouTube there listed as a popular video. It had received 44466 views in the one day since it had been uploaded.

1 dag geleden 44466 keer bekeken



It is a news report on a drink vending machine (a Tokyo fixture) that recommends drinks by taking your picture and doing a little bit of multimedia content analysis that gives it clues as to your age and gender.

In my current situation, age and gender wouldn't have been enough. Rather the system would need information about my internal state -- the camera would have to have noticed the unfocused glaze of my tired eyes. In this situation, internal-state information could be inferred if the system had access to information about my geo-coordinates within the last ten days. Access to a recent history of my sleeping-waking pattern would provide an even better source of evidence.

However, another key bit of information, that would be critical to get to the correct drink would be that at the moment I do not want to be tired. I can't be tired. I don't want something that will relax me -- no chamomile, not yet. I need to work.

The bottom line is clear: barring a system that has access to all that information and the ability to use it in the right way, the Brazilian colleague remains the best source of drink recommendations.

And it looks like the drink is working already, since I have already reached a level of alertness to attempt a blog post.

Authentic, Personalized Travel Recommendation

At the moment a lot of my time and attention is being devoted to Near2Me, which is a concept for a travel recommender developed by a colleague of mine, Luz Caballero, for the PetaMedia Network of Excellence. The cool thing about Near2Me is that in not only makes personalized travel recommendations, but it also focuses on authenticity -- the distinctive "spirit of place", as Luz likes to describe it.

The Near2Me concept tackles the problem of long-tail recommendation, it suggests that you go places where relatively few other people have been: Most of the time we refer to it as Off-The-Beaten-Track. The concept links up with the "Creative Tourism" movement, which holds that travel should be participatory and that travelers should hook up with the living culture of the place as manifested in the lives of the people living there.

How do we get at the authenticity of a place? Well, Luz points out that the act of taking a picture is something like adding a tag to a place. The tag is different than other sorts of tags. For example, it is not like the tags we use on Flikcr, since it doesn't have a lot of semantics. Taking a picture is basically a salience tag, it just says, this came to my attention as interesting. Luz did a study of Flickr where she determined that people who travel, even though they of course take pictures of the "must sees", Notre Dame in Paris, for example, also take pictures of interesting things along the way.

Because the motiviation of the people taking these pictures is something more akin to personal documentation than it is to touristic promotion, they provide a valuable source of information about the authentic, the things that we stumble across when we actually are in a place, along the way to reaching another goal.

The next step is simple, you look at the geotags of the pictures (that little piece of metadata produced by a growing number of cameras that records where the picture was taken) and you use the Amazon principle: people who liked X also liked Y. You look for people who have taken pictures where you have taken pictures and see where else they go. Luz' uses the example of markets: if someone liked the "Borough Market" in London, the system would recommend the "Marche Wilson" in Paris. The technique is called "wormholes" and was developed by Maarten Clements.

At this point, you encounter possibly the most challenging part of long-tail recommendation. You want to give people something new and interested, but none of the place recommendations are going to be obvious since you want to stay away from the bestsellers, the commercial destinations. For this reason, its important to be able to explore recommendations further: OK, the system thinks I should like the Nusantara Museum in Delft, but would I really like it? Why? What is it anyway? What can I see and do there?

Two of the ways that users can explore a place are by browsing pictures and browsing people. The Near2Me concept offers a selection of pictures from the recommended place that are carefully chosen to be both diverse and also representative of a place. Also, it offers a selection of people, who by virtue of the pictures that they take and of the popularity of those pictures, emerge as a sort of community expert for a particular topic. How these algorithms work -- and if they are of use in a working prototype is something in the works targeted for publication in future PetaMedia papers.

On Saturday, I flew back from Dublin after having spent a couple days at DCU. On the way to the airport, the taxi driver asked me where I was going.
"Back to Amsterdam." I said. "I came Wednesday and now I am already going back."
"Oh, Amerdam. Did you bring tulips?"
"No, chocolate." I replied, and then added "The funny thing is, is that I'll probably get some chocolate to bring the other direction as well."
"I'll tell you what you got to get." he said. And then launched into description of how amazing the Baileys Irish Cream chocolates are. I've been to Dublin several times, but I still have to ask people to repeat words sometimes because of the Irish English, which he did quite patiently. In the end the picture emerged that the Baileys Irish Cream chocolates are amazing because they are not just chocolatey, but they offer a real Baileys experience as well.

Last time flying back from Dublin I had seen the Baileys Irish Cream chocolates at the airport and had steered clear -- putting Baileys in chocolate was something, I assumed, they had come up with for the tourists, and that I would do well to avoid it. However, the fact that the taxi driver had no particular reason to tell me this and the fact that he was, himself, to me clearly Irish, I shifted my opinion on the Baileys Irish Cream chocolates from "main stream tourist" to "genuine Irish souvenir". Since I was the one that brought up the chocolate, the taxi driver was certainly not doing product placement. Perhaps Baileys is simply very, very smart with this. But if they are, the Baileys Irish Cream chocolates will still remain authentic to me because the whole story happened at a moment where I was devoting a lot of time and attention to Near2Me and thinking a lot about authenticity of place.

This is Part I of the "Irish Chocolate Discussion", reflections on the conversation I had with a Dublin taxi driver and how that relates to finding things and search systems in general.