Serendipity and Information on the Web.page

http://techcrunch.com/2010/11/27/myth-serendipity/

Also, think about how web interaction can be focused around *experience* instead of information.

Editor’s note: Henry “Hank” Nothhaft, Jr. is the co-founder and CMO of Trapit, a virtual personal assistant for Web content still in private beta that was incubated out of SRI and the CALO project (as was Siri, the conversational search engine bought by Apple).

One of the most interesting concepts to emerge in media and tech lately is that of “serendipity”—showing people what they want even if they didn’t ask for it.

Despite its seemingly ubiquitous invocation, however, the concept of serendipity remains ill-defined and put forth as some vague panacea for a slew of emerging innovations hoping to attract new users in droves.  What is needed is a closer look at what we actually mean when we talk about serendipity.

# From Search to Discovery

Eric Schmidt’s recent remarks about Google as a “Serendipity Engine” (and Facebook’s quick reply), emphasize an important shift in our daily interaction with the Web and how we use it.  Google-driven search provided us with an expectation of finding what we are looking for with increased precision.  But the rise of Facebook’s social relevance algorithms brought about more personalized content discovery based on the human graph—who we know and what they are reading, watching, or passing along.

In fact, I’d argue that we’re seeing the dominant portion of our interaction with Web content shift from search to discovery.

Jeff Jarvis has perhaps most succinctly defined the concept of serendipity, arguing that serendipity is simply “unexpected relevance.”  His explanation opens an entirely new can of worms, however, in the recognition that relevance is relative.

In seeking to achieve serendipity, the individual reader becomes both the target of content delivery mechanisms and the genesis of what that content may be. This is why serendipity is so closely associated with personalization—it requires a high-resolution understanding of the user.

Serendipity and personalization are in fact two sides to the same coin.  Personalization merely acknowledges intimacy, whereas serendipity pretends to have happened on it as if by accident.

Of course serendipity is not, in fact, at all random. In reality, it’s quite scientific. Good serendipity is a slight of hand—it requires deep and granular knowledge, and the fact of its seeming to happen by accident is an artifact of naivety, if anything.

Serendipity is really just an informed calculation based upon any number of our individually unique interests, habits, location, the time and date, and prior knowledge. This level of relevance is, of course, what the emerging personalized Web hopes to achieve for each user, whether for recommendations (GetGlue; Hunch), marketing and ads (Rapleaf; Facebook advertising), or news and content (my company, TrapIt).

Below I run through four different kinds of serendipity—each has its pros and cons. I end by talking about them all taken together, and “the myth of the sweet spot”.

# Editorial Serendipity

Editorial Serendipity is the first and oldest form, the process of combining articles that we know we want to read (the day’s headlines) with unexpected stories (features, profiles, restaurant reviews). Yet the editorial voice and direction of a paper or aggregator is hardly serendipitous; it is a calculation of demographics and readership, whether you’re the New York Times, the Drudge Report, or TechCrunch.

On the plus side here, the human element of editorial serendipity (someone making decisions on what content to deliver) provides an effective flexibility of interest. The downside is that editorial serendipity is delivered by another’s interests, or at best their perception of their audience’s interests. Though the content’s relevance is targeted to a certain demographic of readers, it is a necessarily broad sweep of potential readers, and the level of interest is based on the editors’ perception of what is most in tune with those readers or what she thinks they should be interested in based on her own judgement.

Examples: Newspapers/Magazines, Curated Aggregators

# Social Serendipity

Much of our content discovery now comes from the virtual watercooler of what our social circle is sharing directly online. The social aspect of staying informed with what our friends are discussing is valuable, not only for keeping “in the loop,” but also simply for the notion that what our friends like is parallel to our own interests.

The benefit of social serendipity is that our social groups have always been a primary indicator of how we choose to define ourselves and our interests. If something is important or relevant to our friends, there is a high likelihood that it is also relevant to ourselves, as well. The con is that social serendipity is therefore largely public by necessity, and thus a projection of ourselves we would present to others or like to be seen. The propensity to amplify the echo-chamber of like-mindedness is also exaggerated, whereas the goal of serendipity largely lies in the surprise and delight of unexpected content.

Examples: Facebook, Twitter

# Crowdsourced Serendipity

Bridging the gap between editorial and social serendipity, the notion of crowdsourced relevance really only delivers a broad, lowest-common-denominator level of content discovery. While not without its usefulness to the degree that we want to be aware of what is most popular and most talked about, the trade-off is the lack of personalization.

The pro here is the viral component, which makes up a great deal of our online content-discovery routines. Crowdsourced serendipity provides a tier of distribution in touch with a larger zeitgeist, from trivial cat videos to important broad-based news. The downside is that the lowest common denominator lacks any precision and therefore has limited utility.

Examples: StumbleUpon, Reddit, Digg

# Algorithmic Serendipity

Opposite editorial serendipity, the notion of algorithmic serendipity is the hardest to do well, but the most promising for future innovation. (Bias alert: this is the approach we are trying at TrapIt)

Based-upon any given set of data points, content is personalized to provide both the relevant, need-to-know information of news and content correlating to our interests, with varying degrees of flexibility through both active and passive inputs.

The best aspect of algorithmic serendipity is that it places the user back at the center of defining relevance. Content delivery emanates from the user, whether consciously or in the background based on habit. It also provides for a level of adjustability and fine-tuning based on individualized input and how narrowly or broadly a user may want the information delivered to him.

The con with algorithmic serendipity is that we need to be careful not to completely lose the human element of engagement, no matter how accurate the algorithm is. Of course, the biggest hindrance is that unlike the other forms of serendipity, a finely-tuned algorithmic Serendipity Engine has yet to be effectively realized. Still, it needs to only be the starting point rather than end point of achieving personalized serendipity.

Examples: Genieo, My6Sense, TrapIt

# The Myth of the Sweet Spot

The challenge for any conception of serendipity, regardless of type, is the prevailing notion of a mythical “sweet spot” for users.

In all of the forms of content delivery outlined above, there is a notion that we can hone in on a user’s interests and find the right balance of relevance.  Presenting any such balance as stable or definitive is pure folly. We humans have no “sweet spot”—our interests are evolving and fluid in realtime.

To some extent, this recognition is obvious. Our interests change and evolve over time. Yet for the kind of precision that seeks to provide consistent serendipity in the ways we have been discussing, the indicators need to be equally sensitive.

The content that I want, and better yet, the content that I don’t even know that I want, is an ever-changing proposition based on any number of factors. To achieve that level of sophisticated customization requires a sensitive understanding of context for any proposed “serendipity engine”, both a context of the content and the user.

In the end, relevance is a goal based on context. The impossibility of fully understanding every intricacy of context at any given moment makes achieving the mythical, consistent sweet spot of serendipity impossible. Recognizing that serendipity is a constantly moving target of context, the best we can hope to achieve are fleeting moments relevance.