Recently Watched: A Data Story

Recently watched came up a couple of times in the past as a “nice to have” project, a.k.a. another one for the “maybe never” pile. After all, recency is everywhere. Netflix makes finishing all of House of Cards the default experience through recency. Sony sorts my game library by last played. I’ve been happily opening recent documents since Office 95. Recency can be built entirely client-side, so it’s way simpler and cheaper than most discovery improvements. But it wasn’t clear it’d be valuable for Twitch until I asked our data the right question. How much of our viewership is already on recently watched channels?

Recently watched channels get more viewership than all our existing discovery mechanisms! We launched recency experimentally and the treatment group watched 1.1% more minutes than the control group. Boom, one of Twitch’s most successful experiments ever! Twitch has moved the needle more with huge projects, like launching an iOS app. But we only get to do that once. I’m Danny Hernandez, a data scientist at Twitch, and this is the story of my biggest win in 2015.

The Problem

I almost never watch the majority of content Twitch shows me, but what I do watch is painfully predictable. I primarily watch competitive Dota. When the International or the Summit is on, that’s always the last thing I watched, what I’m currently watching, and the next thing I’m going to watch. I generally don’t watch League, but that’s the most promoted content on Twitch because it’s the most popular. I could get close to the experience I want with following, but that’s a lot of work and I’m often logged out.

Recency is the easiest way to surface the content I engage with. When I imagined Twitch looking like this it felt right.

Recently watched would show me Dota 2 over League of Legends. And I wouldn’t have to do anything. Having a primary game isn’t weird — it’s totally normal.

Recently watched is cheap to build, understandable, and I don’t have to follow or signup to get a Twitch experience tailored to me. I’m not alone in having a primary game. In fact, having a primary game is totally normal. With my increased belief in recency came clarity as to the next step. I needed to find out how much viewership was already on recently watched channels.

Pay Dirt

It only took a day to build up Recency in Redshift, AWS’s data warehouse [1]. Half of what people watch are channels they recently watched [2]. Boom!

52% of viewership is on the 5 most recently watched channels that are currently live. I only counted a channel as recently watched after a user watches at least 5 minutes. Watching for less than five minutes is actually a kind of negative signal. To make analysis easy I define viewership in terms of views longer than 5 minutes [2]. I was onto something big. Recency was already huge, and it wasn’t surfaced on Twitch at all. As I dug in, the recency story only got better. My analysis was based on a six-week sample, so many viewers didn’t have sufficient history for recency to really shine [3]. Once a viewer built a sufficient viewing history, that 50% shot up to 70%.

Only 75% of viewership in my dataset was by users with an average of 5 live recently watched channels[3]. For these users 70% of viewership is on the 5 most recently watched channels. More users would definitely have sufficient history to surface recent channels in a longer time slice.

It was time to give context to the power of recency. How does recency fare when pitted against existing Twitch discovery features?

Going Deep

Recency is powerful, but to understand just how powerful, we need context. How does recency stack up against our most prominent discovery features? Channels you follow and popular channels in a given game are the two primary ways viewers discover channels now, so lets start there.

30% of viewership on Twitch is on user’s top 5 followed channels. In order to have five live followed channels, users need 40+ follows.

Surprised? Recency outperforms following because it helps more than twice as many people. Recency helps everyone eventually, because it requires no effort from the viewer. Following must be useful though, right? It’s the foundation of Twitter… Following does in fact do wonders, but only for our power users (users with 5 live followed channels).

Following is incredibly powerful once a user has 5 live followed channels. A user needs 40 follows to get to this stage.

Power users make up 30% of viewership, and following predicts 90% of what they’ll watch. So power users on Twitch are sitting pretty when looking for content. They have following and it’s incredibly useful for them. But 30% of our viewership is by logged out and as such following nothing.

Lets compare recency to the primary way logged out users find channels to watch — popular channels. No surprise here, recency wins!

Recent channels also strongly outperform popular channels.

Recently watched isn’t just strong, it’s strong when compared to the incumbents. Recently watched also requires less space than popular, so it’s easy to integrate. The most recent live channel is almost 4x as useful as the most popular channel, so it’s perfect for limited, highly visible real estate. One last nugget: recent games perform at a similar level to recent channels.

In Conclusion, Recency is Dope

I shared everything above internally. Within a week, planning started for an experiment to evaluate the impact of recency in the wild. No begging, favors, or sweet talking was required. The findings were convincing enough on their own. It was super satisfying to take something from idea to analysis and then finally to “we’re working on this right now.”

The treatment group for the recency experiment watched 1.1% more minutes over the two week experiment. Logged out users benefited the most. Their minutes watched increased by 1.5%. If they used the feature they watched 3% more and showed up 2% more days. Recent channels make logged-out browsing much closer to logged-in browsing by providing a substitute for followed channels.

This change was evaluated and rolled out based on short term success. But recency could have important long term effects for communities. I found that broadcasters that average 3000 concurrents had almost twice the retention of those that average 150. I tried to control for time spent broadcasting, so I believe the issue is that it’s hard to run into a small broadcaster. It’s easy to find popular broadcasters on Twitch without actively seeking them, but it takes effort to find small broadcasters. It’s been found that repeated, unplanned interactions are critical to building close friendships. Twitch doesn’t need to rely on serendipity. Recency will help viewers casually bump into broadcasters regardless of popularity.

Recently watched is simple, cheap, and strongly predictive for Twitch and it can be implemented entirely client-side. If you work on a product, consider adding recency. Keep in mind recency is about helping you get back to where you left off, not about finding new things you’d be interested in. New things you might be interested in is a separate, harder problem. If work like this is your jam, consider joining the Twitch Science Team.

Footnotes

1. I built recency in redshift with Window Functions. Window functions allow you to operate on sets of rows in a table based on some kind of ordering and partitioning. The lag window function allows you to enrich a row with data from the next row. I partitioned my data into users and sorted a users viewing history in time. I ran through this table once enriching it with whether or not this was the first viewing of the channel within 1, 2, and 5 plays. Those would all be recent cache hits. I ran through the table a second time looking at only non duplicate plays. Again I enriched the rows with whether or not this was the first viewing of the channel within 1, 2 and 5 de-duplicated plays. Those would also all be recent cache hits. Boom Recency.

2. I only count a channel as recently watched after a viewer watches at least 5 minutes. Watching for less than five minutes is actually a kind of negative signal. To make analysis easy I define viewership in terms of five minute plays. A whopping 91% of Twitch’s minutes watched are on video views longer than 5 minutes.

3. Conservatively I assumed a user needed a history of 5 plays to have one live channel on average. That was much easier than building a sorted live recently watched channel list in time in sql. I based my assumption on the fact that 1 out of every 8 followed channels is live at a given time.