This study should be an embarrassment to a company that cultivates a reputation of knowing something about statistics. It's like the last 500 years of statisticians working on survival analysis didn't happen.
Their conclusions are not supported by the data, the study methodology is flawed, and their "average of averages" metric would have gotten me an F from my 8th grade physics teacher.
The key methodology problem is this: "A hooked episode was defined when 70% of viewers who watched that episode went on to complete season one." What you are actually want to know is after which episode the retention curve flattens out. With their methodology they'd claim a "hooked episode" even in case show's retention followed exponential decay (constant % drop after each episode) - meaning there is no "hooked" episode at all, and retention curve never flattens out.
"Hooked episodes were first identified by country, then averaged to create the global hooked episode." This is just awful. Again, high-school physics - don't average averages. I understand what they are trying to do but this is not how you do it.
And to add insult to the injury they stuck a House of Cards spoiler in the first sentence.
Except they use the data to make stupid conclusions:
"However, in our research of more than 20 shows across 16 markets, we found that no one was ever hooked on the pilot. This gives us confidence that giving our members all episodes at once is more aligned with how fans are made."
I don't see how the study sheds light into "causality," i.e. "how fans are made."
Since netflix does not need to sell commercials, there is no incentive problem with binging. A network, however, needs to keep your attention every Wednesday evening to sell their commercial slots, so they have to pace their material episode by episode.
Well I was hooked on Dare Devil, The IT Crowd, House of Cards and Firefly on the first episode, but these shows pander to me beyond just plot, I'm sure you can detect the theme for most of them. I'm sure there is a key point for most shows that most who watch up to that point end up watching the whole thing. BSG I remember took 10-12 episodes before I really got into it, and I only went that far because people wouldn't shut up about how good it was. There are also shows I really liked but either dragged on or I just lost interest in or was just frustrated by the characters (The Killing, Buffy, etc)
This is certainly true for me. If I have any inkling I'll like a show I don't even let it infect my opinion if I dislike the pilot, as that's normally he case. Heck, for shows that have been around a while sometimes I don't even care for the first season but keep watching - some of those have turned into my favorites
The Wire is the only show coming to mind where I was completely sold by he first ep.
I usually watch shows in a similar way, which is also why I try to wait until all of the first season is available on-demand before starting a show. I'm also fairly cautious with shows and waiting until the full season has run allows a lot of critical opinion to have already surfaced.
Regarding the Wire, I've watched the series three times now and I'm still not sure the first episode is a good hook, or I even started to get hooked. What was a big surprise to me the third time through is how I still get transfixed on season 4.
One case where I was hooked by the third episode but ultimately gave up on is Orange is the New Black. I found most of the first season fairly compelling, but by the end, it was apparent the show was going down a road that I wasn't interested in. I would have preferred to keep the series relatively realistic, run a season or two in prison and then focus the end on the transition back. Instead, they kept the main character in prison, seemingly indefinitely, and started putting a lot more focus on the other characters. I skipped around season three a little bit and was deeply disappointed at how it treated meaty topics like private prison profiteering so flippantly. I suppose The Wire did set the bar unreasonably high for dealing with those sorts of topics.
There are many shows that for whatever reason I don't start watching initially but then enough people I trust like them that I'll watch. Those are the ones that fit the pattern we're discussing.
There are many other shows which I don't have high hopes for but start watching just because they're on. For those I won't give it the full season but I also don't judge just by the first few eps either. These are the sorts of shows that often don't make it past a season or two anyways so it's a bit academic :)
I like OitNB but have the same issues you state. It's a fine show but not great by any means. The sort of show I put on while I'm reading blogs and such.
The user may potentially watch everything they want within a couple weeks and then unsubscribe. They would make more money if they were forced to be subscribed for longer in order to finish shows.
>forced to be subscribed for longer in order to finish shows.
Which is exactly how HBO has been able to keep me subscribed for months on end. I thought I'd cancel it after Silicon Valley finished. HBO comes out with good content every week to keep me hooked.
"Netflix has no plans to use Big Data to rejigger the way TV shows get made"
It's hard to believe data like this won't end up being used to reverse engineer some aspects of their content, especially when they are pumping out dozens of shows. Easy to experiment with some of them.
Amazon already has users voting on solo pilots and it would be very surprising if they don't end up leveraging their analytics for creative.
Here's what I'd like to know. Do they have any data on the optimum length of a plot arc for a shows? There are a number of shows I've gotten hooked and then around about season 5-7 I've dropped because it became obvious they were just milking it and inventing gratuitous plot extensions instead of ending it cleanly.
Some shows can go for really long runs because the plot arcs tend to be 1-3 episodes in length. The Law and Order style of show. Anything that goes multiple seasons in length just seems to lose me after at most 2 plot arcs. Maybe I just get burnt out?
> There are a number of shows I've gotten hooked and then around about season 5-7 I've dropped because it became obvious they were just milking it and inventing gratuitous plot extensions instead of ending it cleanly.
This is what I'm worried about with 'House of Cards.' It feels like they might have 1-2 good seasons left, but unless Frank suddenly decides to deceive his way into becoming world dictator, I'm convinced that they need to end the show in fewer seasons than it already has.
I wonder if there had ever been a show in the history of television in which 70% of viewers who watched the pilot made it to the finale. Fortunately, the article states Netflix has no plans to "rejigger" (i like that word, plan on using it more) its method of greenlighting shows. Which probably, despite the enormous quantity of data they collect, still resembles something akin to that scene in "Network," where they consider doing a docu-drama about the Symbionese Liberation Army.
You can count me as one of the 30% that never makes it past the first ten minutes of even the most acclaimed serial content. I just like movies. But out of curiosity I did watch the extended trailer of the 500 Startups reality show "Bazillion Dollar Club." There is a scene where a team is finalizing a prototype with its sourced design consultants. Its all dudes and they select a shape that is, well, let's just say suggestively masculine in its proportion. They decide they need to counterbalance their opinion with the female perspective and invite the ladies on staff to give their opinion. The women choose a completely different design. Soft, with rounded curves. But most importantly they have a visceral negative reaction to the men's choice. But the startup leadership completely ignores what I would consider a very important data point in favour of an arbitrary metric. That "70% of men make electronic purchasing decisions for their households." Completely ignoring the fact that generally, men don't really care about the appearance of their gadgets, just that they function properly. And that women generally tend to make the design choices at home. Obviously a very broad generalization but still.
The lesson here is about letting stats influence faulty reasoning while ignoring intuition. It doesn't appear that Netflix is allowing data to dictate programming. Yet.
I suppose they measure it by looking at the episode at which users continue to watch the whole series, in contrast to those who stopped viewing at an earlier episode? There is so much more you could get out of this:
(i) What's the drop off rate for viewers over episode number? For example, I stopped viewing quite a lot of these shows right before other viewers got hooked (Marco Polo: episode 2, sense 8: episode 2, that superannoying lesbian scene, Arrow: episode 3 or 4)
(ii) What's the interval distribution between successive episode viewings, does it differ between shows, is it predictive of a tendency to stay with the show (i.e., how bingy is the show, if that's a word)?
(iii) Age and gender distributions for drop off rates
And so much more. Any way of getting my hands on the data?
DO YOU KNOW WHEN YOU WERE HOOKED? NETFLIX DOES
Netflix Unveils When Fandom Begins For Some of Todays Most Popular Series
Sep 23, 2015
Los Gatos, Calif., September 23, 2015 -- It may have taken Walter White nearly an entire season to become Heisenberg and Frank Underwood 13 episodes to become VP (spoiler alert!), but it turns out fans committed to these series long before those plot twists unfolded. Hint: it wasnt in the pilot episode.
Netflix analyzed its global streaming data* across the inaugural seasons of some of todays most popular shows - both Netflix original series and shows that premiered on other networks - looking for signals that pointed to when viewers became hooked. It turns out that when commercial breaks and appointment viewing are stripped away and consumers can watch an entire season as they choose, you can see fandom emerge. That is, 70% of viewers who watched the hooked episode went on to complete season one or more poetically, when members were hooked and there was no turning back.
“Given the precious nature of primetime slots on traditional TV, a series pilot is arguably the most important point in the life of the show,” said Ted Sarandos, Chief Content Officer for Netflix. “However, in our research of more than 20 shows across 16 markets, we found that no one was ever hooked on the pilot. This gives us confidence that giving our members all episodes at once is more aligned with how fans are made.”
While the data identified the hooked episode, it was shy on pinpointing exact moments, but we have a few ideas of our own to help jog your memory... For starters, in Breaking Bad it may have taken the flip of a coin to decide whether Jesse or Walt would put the finishing blow on Krazy 8, but when the decrepit heap of a former drug dealer rains down from Jesse’s ceiling, theres no denying viewers would stay to see how the season cleaned up (episode 2). Speaking of messes, Crazy Eyes drops both poems and fluids in her roller coaster romance with Piper in Orange is the New Black, but it was likely the throw of a pie to defend her (then) bae’s honor that had members asking for seconds (episode 3). For Dexter another episode equals another body, this time courtesy of the “Ice Truck Killer,” but our money’s on Dexter’s trip down memory lane reliving his inaugural kill that was the real tipping point - after all, fans never forget the first time (episode 3).
“There’s a unique sense of intimacy with creating a show for Netflix. Knowing you have an audience's undivided attention and that in essence, they are letting these characters in their home, we unfolded storylines at a more natural pace,” said Marta Kauffman. “In episode four, we see Grace and Frankie having no choice but to confront their fear, anger and uncertainty head on, which to me as a creator was a nice turning point to shift the narrative to focus on the future instead of the past; it is nice to know viewers were there right along with us.”
While around the world the hooked episode was relatively consistent, slight geographic differences did present themselves. The Dutch, for instance, tend to fall in love with series the fastest, getting hooked one episode ahead of most countries irrespective of the show. Germans showed early fandom for Arrow whereas France fell first for How I Met Your Mother. In Better Call Saul, Jimmy McGill won Brazilians over one episode quicker than Mexicans. And Down Under, viewers prove to hold out longer across the board, with members in Australia and New Zealand getting hooked one to two episodes later than the rest of the world on almost every show. Despite these differences, the hooked moment had no correlation to audience size or attrition, regardless of show, episode number or country.
Methodology:
The data in this research was pulled from accounts who started watching season one of the selected series between January 2015 - July 2015 in Brazil, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, Mexico, Netherlands, Norway, Sweden, UK and US and between April 2015 - July 2015 for Australia and New Zealand. A hooked episode was defined when 70% of viewers who watched that episode went on to complete season one. Hooked episodes were first identified by country, then averaged to create the global hooked episode. The hooked episode had no correlation to total viewership numbers or attrition.
Denotes shows where for one or more countries, the show was unavailable to watch on Netflix and therefore the average is comprised of data from less than 16 countries
*The Netflix research didnt indicate exact plot points, but it did confirm episodes.
There's some side of me that's unhappy to see stats (even as benign as this) being considered. It's not really the idea that statistics will influence content as much as it is about who and how it will influence it.
I've been reading "A Song of Ice and Fire" (the source books for "Game of Thrones"), like a lot of people, for years. That is, I've been reading a book enjoyably for a couple of months and then waiting 5 years for the next one. It's an incredibly intricate and big story spanning a lot of character and geography. It's very big. The author, is obviously a special talent with incredible story telling ability and character perspectives. The talent for getting into characters' heads makes me think he must be a very emotionally invested artist.
Anyway, among all the characters, story, best seller pressure, TV show, and such writing the books takes a long time. He's got a Ford Prefect like relationship with deadlines.
Now.. in my (naive) mind, that's as it should be. A chaotic, burdensome process that (at the end, sometimes) produces art.
TV shows and films are the art of our times. This is not a statement about quality, just about the reach, influence and such. The people who make films are famous and admired. Everyone gets the references and allusions. Almost everyone indulges in it, grows up with it. A lot of people here probably got my Douglas Adams reference, but probably fewer than would have gotten a reference to "Morpheus," even in this biased crowd.
But, movies and television shows are made by big groups of people, money, risk-capital, adult supervision. I'm OK with statistical information about pace, plot or whatnot eventually permeating through the arts. That probably makes the arts better. I dislike the idea of it being used as a mallet in the proverbial committee designing the thing.
I like sausage. I prefer sausage to be made in a nice way. If you insist on both making it in a way I don't want to see and constantly reminding me that I should not look into how it is made… I'll have some beef stew instead.
Seems a bit weak - I don't think you can really drill down onto one episode as one that 'hooks' a viewer, rather than all the subsequent episodes being excellent. At least, this article didn't convince me.
Think of the season like a funnel. What's the drop off for each episode? What's the likelihood if they've made it this far they'll watch the rest of the season?
...right, but then I'd rather see the funnel than which episode gets you to 70%, because I don't think without that context you can really say anything about the episode.
Highly unlikely but would be interesting if Amazon releases the same data for ebooks. They might already be doing stuff like showing those pages in the preview that have the highest chance of getting the reader hooked.
Kobo evidently are doing some analysis on completion rates, and published a report last year aimed at publishers using analysis based on similar metrics:
Nice find. Yes raw data will be more useful. Especially, with Amazon's Pay-Per-Page for Kindle Unlimited, if such data is available, it will be interesting to see how authors reverse engineer this process to achieve maximum page views (and completion rates)
This really would be interesting. I'd like to see data on overall completion % for ebooks. Assessing those books which tend to be completed once started would help identify the more interesting books.
Combined with top selling book lists, this would provide a good filter to decide on potential new reads and could form the heart of improved book recommendation services.
I am absolutely not at all whatsoever impressed with Netflix' data science capabilities.
They waste one row for:
- watch it again ...
- they suggest stuff based on what I added to my list, but didn't watch yet
- they suggest movies based on what I started to watch but didn't finish - as if that wasn't an indicator for that I didn't like it
- ...
All those companies suggesting on varies channels how nifty and smart they work with their big data - and at the bottom line they just fail on actually improving something.
Do you think that those design decisions are not backed by data demonstrating better engagement levels versus other interface layouts? I'm not saying that their interface is perfect, but I don't see how you can make sweeping claims about their data science capabilities based on your one data point of disliking their interface.
I think at some point people assume others are more competent than they actually are. I mean, honestly, here you are defending Netflix based on nothing more than stereotypes and projecting competence into them--when the linked PR story is so ridiculously methodologically flawed that a rational observer would seriously question their basic mathematical and psychological literacy.
If you don't like Netflix's ratings recommendations, you might like to try MovieLens.org. They let you choose between different algorithms & provide some additional stats (eg your average rating per genre, unusual likes & dislikes, distribution of ratings by decade).
Cool! But at some point, providing stats yourself is just fudging the algorithm. Why not just browse those categories yourself, instead of diddle an algorithm and seeing if it predicted correctly? Except maybe that's fun too.
Their conclusions are not supported by the data, the study methodology is flawed, and their "average of averages" metric would have gotten me an F from my 8th grade physics teacher.
The key methodology problem is this: "A hooked episode was defined when 70% of viewers who watched that episode went on to complete season one." What you are actually want to know is after which episode the retention curve flattens out. With their methodology they'd claim a "hooked episode" even in case show's retention followed exponential decay (constant % drop after each episode) - meaning there is no "hooked" episode at all, and retention curve never flattens out.
"Hooked episodes were first identified by country, then averaged to create the global hooked episode." This is just awful. Again, high-school physics - don't average averages. I understand what they are trying to do but this is not how you do it.
And to add insult to the injury they stuck a House of Cards spoiler in the first sentence.