Over the past few months, Netflix has dared some of its employees in its Los Gatos offices with a special kind of challenge: Two TVs mounted side-by-side were playing the same TV show episode. One was coming straight from Netflix’s existing service, the other was based on a new bandwidth-saving technology that the company has been working on for four years. Anyone capable of pointing out the difference could win a bottle of champagne. But in the end, even eagle-eyed employees had to give up, and the prize went unclaimed.
Encouraged by these results, as well as months of additional testing, Netflix now has begun to embark on one of the biggest changes to its streaming technology since it launched its online video service in 2007. If all goes according to plan, the switch could help consumers get better-looking streams while also saving up to 20 percent of data — which is significant in North America, where Netflix usage single-handedly accounts for more than a third of all data consumed during peak times, and an even bigger deal in all those countries with relatively slow internet speeds that the company is looking to enter in 2016.
‘My Little Pony’ Is Not the Same as ‘The Avengers’
Netflix has been working on this new technology since 2011, when members of its video algorithms team realized that they had gotten it all wrong. Like practically everyone else in the online video world, Netflix had been preparing its video files for streaming based on the bandwidth available to consumers. Some Netflix subscribers were accessing the service with slow DSL connections, others had faster cable connections, and a lucky few were already online with super-fast fiber speeds.
Popular on Variety
Based on these use cases, Netflix’s video algorithms team had developed a number of quality levels, or recipes, as they’re called in the world of video encoding. Each video file on Netflix’s servers was being prepared with these same recipes to make multiple versions necessary to serve users at different speeds. At the lowest end was a file encoded with a bitrate of 235 kbps, which would work even on very slow connections, but also only deliver a resolution of 320 by 240 pixels. Somewhere in the middle was a 1750 kbps file for a resolution of 1280 by 720, and the best quality was a 5800 kbps version for a great-looking 1080p experience.
Netflix’s service has been dynamically delivering these versions based on a consumer’s bandwidth needs, which is why the quality of a stream occasionally shifts in the middle of a binge-watching session. But across its entire catalog of movies and TV shows, the company has been using the same rules — which didn’t really make sense. “You shouldn’t allocate the same amount of bits for ‘My Little Pony’ as for ‘The Avengers,’” explained Netflix video algorithms manager Anne Aaron.
 |
| To a computer in charge of encoding, ‘My Little Pony’ just isn’t that complex. Courtesy of Hasbro / Netflix |
That’s because a show like “My Little Pony” is not at all like the ‘Avengers’ — especially to a computer in charge of preparing media files for streaming. Animated shows like “My Little Pony” can be reproduced with relatively little data. An animated sky, for example, tends to be filled with the same shade of blue, and an animated pony that just stands around talking isn’t very complex to a computer either. The “Avengers” on the other hand is full of fast-paced action, which plays out in front of cityscapes and other environments with lots of visual details. And, of course, explosions, rubble, smoke, and lots of it. Or as video engineers like to call it: noise. “Noise is hard to encode,” said Aaron.
It’s a much more complex movie, at least from a purely visual perspective. This means that Netflix’s encoding servers have to work a lot harder to get it down to the same file sizes as they do with “My Little Pony,” leading to more compression and possibly more visible artifacts at low bitrates. That’s why in 2011, Netflix’s video algorithm engineers realized that they shouldn’t apply the same encoding rules to these two very different titles. “A one-size-fits-all model doesn’t give you the most optimal quality,” said Aaron.
Instead, they decided that each title should get its own set of rules. This allows the company to stream visually simple videos like “My Little Pony” in a 1080p resolution with a bitrate of just 1.5 Mbps. In other words: Even someone with a very slow broadband or mobile internet connection can watch the animated show in full HD quality under the new approach. Previously, the same consumer would have just been able to watch the show with a resolution of 720*480, and still used more data.
But Netflix’s new per title approach doesn’t just improve things for animated TV shows. Fans of other fare are also set to benefit, and see significant bandwidth savings when streaming the highest quality 1080p video. Netflix’s video algorithms team set up a test for Variety in its offices in Los Gatos last week, streaming two episodes of “Orange is the New Black” in 1080p on two TVs mounted side-by-side, similar to the tests the company has been doing with its employees. The images on both TVs looked virtually identical — but one streamed with 5800 kbps, using Netflix’s legacy encoding scheme, whereas the other displayed the show with 4640 kbps. The difference? 20 percent in bandwidth savings.
Each episode of a TV show can have different settings
Over the past couple of years, Netflix’s video algorithm engineers have worked on perfecting this more flexible approach towards video encoding. Initially, they assumed that they would just have to categorize all of their titles to develop new encoding rules for animated shows, action movies, slow indie dramas and so forth. But they quickly realized that no two indie movies are alike, and that there can even be considerable visual differences within a season of the same TV show. “Each episode could be very different,” said Aaron. The result is a true title-by-title approach, where every single movie and TV show episode gets its own encoding settings.
SEE MORE: Why Netflix Is Pushing ‘Breaking Bad,’ ‘House of Cards’ in 4K Ultra HD
In recent months, Netflix has been silently testing the new encoding scheme by sending out streams with new and improved bitrates and resolutions to randomly selected customers. The company did keep a close eye on completion rates and streaming duration, but it also wanted to make sure that all devices out in the field were able to deal with the new bitrates.
Separately, Netflix has been testing the new and improved videos in person both with its own employees as well as with a number of customers. And earlier this month, Netflix quietly added a first batch of videos encoded with the new per-title approach to its catalog. Beginning with some of the most popular videos, the company aims to have a thousand re-encoded titles in its catalog by the holidays, and complete the entire process by the end of Q1 2016.
Movies are re-encoded while you’re asleep
This whole endeavor couldn’t have been possible without some considerable technical advances. First of all, Netflix needed a good way to analyze the visual quality of a video file when encoded with different settings. The company doesn’t really talk about the size of its catalog, but it’s fair to assume that the entirety of its catalog, including all TV show episodes, movies and trailers across all of its markets around the world is in the six digits, and new titles are constantly added.
It’s impossible for Netflix’s employees to visually inspect every single one of these videos in all available resolutions and bitrates. That’s why Netflix partnered with researchers at the University of Southern California, the University of Nantes and the UT Austin to develop technology to automate this process.
But knowing which quality would be best for a video and actually redoing Netflix’s entire catalog are still two very different things. The company has been using two tricks to make this approach even more efficient and prepare itself for the massive re-encoding project: It cuts every title into numerous slices, making it possible for multiple servers to crunch away on it at the same time, significantly speeding up encoding.
Netflix has long used cloud computing instead of its own data centers to run its service, and the encoding is also done on Amazon’s web services. However, the sheer amount of computing power necessary for a redo of its entire catalog is still significant, even in a cloud world where capacity can be added on demand. That’s why Netflix is shifting around its encoding jobs on the server instances it rents from Amazon to make use of any idle time. Netflix’s Amazon servers may help you to binge on “Jessica Jones” in the evening, but at night, when everyone is asleep, those very same servers are busy re-encoding the company’s catalog.
New markets require better-looking videos at lower speeds
Netflix’s new per-title approach is good news for its subscribers, who now get to watch many of the service’s videos with a better quality while saving bandwidth at the same time. But it could be an even bigger deal for phone and cable companies and the internet at large.
Netflix is now responsible for 37 percent of all internet traffic going into people’s homes in North America during those peak times when everyone is in front of their TVs, according to recent data from Sandvine. Reducing that amount of data by up to 20 percent could help to alleviate internet bottlenecks. “We want to be good stewards of the internet,” said Aaron.
Netflix’s impact on the internet has been a bit of a touchy subject in the past. Cable companies like Comcast and Time Warner Cable wanted Netflix to reimburse them for sending so much traffic to their customers. Netflix argued that this would amount to double billing because those customers are already paying for their service. The internet companies disagreed, and a public stand-off temporarily led to Netflix speeds going down for many customers. In the end, Netflix gave in and agreed to pay up. But as the company is expanding around the world, similar issues are likely to pop up again. Anything that helps Netflix to slow down the growth of its traffic is likely going to make things a lot easier for the company.
SEE MORE: Netflix Wants the World: Can It Really Expand to 200 Countries in 2 Years?
What’s more, many of Netflix’s future markets have much slower internet speeds. Netflix has said that it wants to be in all countries around the world by the end of 2016. Expanding to India, Africa and the Middle East in many cases means taking a mobile-first approach, or dealing with much slower wired internet speeds. Both will benefit significantly from the new encoding scheme, which will allow Netflix to deliver better-looking videos at slower speeds.
Once Netflix’s video algorithms team is done with re-encoding the existing catalog, it already has a number of new challenges ahead. There is 4K, and there are a number of other ways to improve image quality, including higher frame rates and HDR.
And then there is another crazy idea that could require the company to re-encode the entire catalog all over again: After finding the best setting for each single video, Aaron’s team is now thinking about even encoding each scene of a movie or TV show with different settings to account for higher information density during fight scenes and lower demands during slow moments of introspection.
For Netflix employees, this could mean more chances to finally win that unclaimed champagne bottle.